[spark] supports converting some SparkPredicate to Paimon between LeafPredicate

[steFaiz]

Purpose

Currently, spark will convert Between predicate to the composition of lessOrEqual and GreaterOrEqual, this PR is about to recognize this pattern, converting some And CompoundedPredicate to a single Between LeafPredicate.

Linked issue: none

Tests

Please see org.apache.paimon.spark.sql.SparkV2FilterConverterTestBase

API and Format

No changes.

Documentation

No changes.

Generative AI tooling

This PR is fully hand-written.

[steFaiz]

This PR also rewrite data filters in PaimonBatchScanBuilder, trying to merge candidates filters into Between LeafPredicate.
This is because Spark will split Between into two separated LessOfEqual and GreaterOrEqual SparkPredicates (Not a AND predicate with children).

[JingsongLi]

We could have a PredicateRewrite in core. Maybe invoked in ReadBuilder.withFilter.

[steFaiz]

@JingsongLi Thanks for your advise! I've refactored it. But it's worthy to note that implementing this logic in core is much more complicated than in spark. We have to consider broader scenarios, for example:

1. recursive: OR(AND(a >= 1, a <= 10, a is not null), b > 10, ... )
2. chained AND: AND(a >= 1, AND(a <= 10, b > 10)) (this could be converted to BETWEEN(a, 1, 10))

and more. maybe I've missed some scenario.

Please take a look and i'm pleasant to improve my code and fix potential bugs.