[fix](ann range search) range search prepare failed on NULL literal

[zhiqiang-hhhh]

What problem does this PR solve?

fix Handle nullable literal gracefully in ANN range search

Refactor prepare_ann_range_search to check the distance function first,
then validate the literal. This prevents crashes when the right-side literal
has a nullable type.

Previously, when a nullable literal (e.g., from a scalar subquery result)
was passed to ANN range search:

• Debug mode: threw an exception inside #ifndef NDEBUG block
• Release mode: crashed in assert_cast<ColumnFloat64*> because the actual
  column type was ColumnNullable

Now the function gracefully marks the expression as unsuitable for ANN
index optimization instead of crashing, allowing the query to fall back
to normal execution path.

Changes:

1. Reorder validation: check distance function before checking literal
2. Replace debug-only exception with mark_unsuitable() for nullable literals
3. Add consistency check between Cast expression and literal type
4. Improve code structure with clear step comments

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[Thearas]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[zhiqiang-hhhh]

run buildall

[doris-robot]

TPC-H: Total hot run time: 30181 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d9147abefbfe107f7db68f9787d0c3bc26612e7f, data reload: false

------ Round 1 ----------------------------------
q1	17628	4533	4281	4281
q2	2019	361	228	228
q3	10153	1281	731	731
q4	10202	785	304	304
q5	7539	2144	1905	1905
q6	198	177	146	146
q7	892	716	616	616
q8	9287	1365	1128	1128
q9	4730	4702	4575	4575
q10	6811	1917	1551	1551
q11	524	294	290	290
q12	334	379	221	221
q13	17788	3983	3227	3227
q14	235	234	221	221
q15	912	807	793	793
q16	688	684	620	620
q17	693	803	536	536
q18	6407	5935	5808	5808
q19	1095	973	593	593
q20	499	487	379	379
q21	2492	1814	1750	1750
q22	365	338	278	278
Total cold run time: 101491 ms
Total hot run time: 30181 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4355	4336	4361	4336
q2	262	329	250	250
q3	2068	2651	2272	2272
q4	1338	1708	1277	1277
q5	4387	4265	4315	4265
q6	217	181	140	140
q7	1880	1844	1725	1725
q8	2506	2792	2543	2543
q9	7664	7445	7305	7305
q10	2925	3009	2686	2686
q11	550	468	455	455
q12	683	765	631	631
q13	3806	4465	3410	3410
q14	318	334	364	334
q15	859	838	834	834
q16	667	756	682	682
q17	1184	1300	1418	1300
q18	8084	8101	8053	8053
q19	914	864	814	814
q20	2053	2129	2213	2129
q21	4714	4404	4189	4189
q22	586	559	504	504
Total cold run time: 52020 ms
Total hot run time: 50134 ms

[doris-robot]

ClickBench: Total hot run time: 28.44 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d9147abefbfe107f7db68f9787d0c3bc26612e7f, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.11	0.11
query5	0.26	0.26	0.25
query6	1.16	0.68	0.69
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.56	0.51	0.49
query10	0.56	0.55	0.56
query11	0.14	0.09	0.10
query12	0.14	0.10	0.11
query13	0.63	0.62	0.62
query14	1.07	1.06	1.05
query15	0.87	0.88	0.88
query16	0.41	0.40	0.39
query17	1.15	1.14	1.11
query18	0.22	0.21	0.21
query19	2.05	1.99	2.06
query20	0.02	0.01	0.02
query21	15.39	0.29	0.15
query22	5.36	0.05	0.04
query23	16.09	0.29	0.10
query24	2.41	0.35	0.64
query25	0.12	0.07	0.06
query26	0.13	0.13	0.15
query27	0.10	0.06	0.05
query28	4.70	1.15	0.97
query29	12.59	3.91	3.16
query30	0.27	0.13	0.11
query31	2.84	0.65	0.40
query32	3.23	0.58	0.49
query33	3.22	3.20	3.26
query34	16.14	5.39	4.79
query35	4.81	4.82	4.79
query36	0.66	0.52	0.49
query37	0.11	0.07	0.07
query38	0.08	0.04	0.04
query39	0.04	0.03	0.04
query40	0.19	0.17	0.16
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.04	0.04
Total cold run time: 99.97 s
Total hot run time: 28.44 s

[hello-stephen]

BE UT Coverage Report

Increment line coverage 58.54% (24/41) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.65% (19423/36894)
Line Coverage	36.14% (180684/499966)
Region Coverage	32.44% (139891/431257)
Branch Coverage	33.49% (60651/181093)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 78.05% (32/41) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.64% (25902/36155)
Line Coverage	54.23% (270447/498742)
Region Coverage	51.43% (224057/435641)
Branch Coverage	53.06% (96459/181801)