Skip to content

[feat](catalog) Support include_table_list and lower_case_database_names for external catalog#60580

Open
morningman wants to merge 5 commits intoapache:masterfrom
morningman:include_table_list
Open

[feat](catalog) Support include_table_list and lower_case_database_names for external catalog#60580
morningman wants to merge 5 commits intoapache:masterfrom
morningman:include_table_list

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Feb 6, 2026

What problem does this PR solve?

Add include_table_list catalog property to allow users to specify an
explicit list of tables (in "db.tbl" format) to include when listing
tables from external catalogs such as HMS. When set, only the specified
tables are returned instead of fetching the full table list from the
remote metastore, which can significantly reduce metadata overhead for
catalogs with a large number of tables.

Sometimes calling getAllTables from HMS will timeout, can use this to avoid.

Add lower_case_database_names catalog property with MySQL-compatible
semantics:

  • 0: case-sensitive (default)
  • 1: database names are stored as lowercase
  • 2: database names are compared case-insensitively

Also rename only_test_lower_case_table_names to lower_case_table_names
(with backward compatibility), and move shared property constants
(INCLUDE_DATABASE_LIST, EXCLUDE_DATABASE_LIST, LOWER_CASE_META_NAMES,
META_NAMES_MAPPING) from Resource to ExternalCatalog where they
logically belong.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 6, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman morningman changed the title [feat](catalog) support include_table_list [feat](catalog) Support include_table_list and lower_case_database_names for external catalog Feb 6, 2026
@morningman
Copy link
Contributor Author

run buildall

1 similar comment
@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 30010 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3742b8f748100bcdfd53efb48c68de63b7b98863, data reload: false

------ Round 1 ----------------------------------
q1	17624	4460	4296	4296
q2	2039	343	225	225
q3	10177	1275	720	720
q4	10187	777	303	303
q5	7543	2133	1926	1926
q6	192	176	145	145
q7	884	750	616	616
q8	9265	1314	1096	1096
q9	4638	4511	4623	4511
q10	6854	1920	1540	1540
q11	520	305	280	280
q12	330	419	219	219
q13	17769	4026	3194	3194
q14	222	237	219	219
q15	878	803	803	803
q16	688	668	635	635
q17	686	801	575	575
q18	6358	5777	5696	5696
q19	1263	998	626	626
q20	508	495	383	383
q21	2463	1840	1723	1723
q22	355	324	279	279
Total cold run time: 101443 ms
Total hot run time: 30010 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4331	4304	4327	4304
q2	251	339	251	251
q3	2080	2631	2227	2227
q4	1374	1695	1248	1248
q5	4294	4226	4156	4156
q6	216	179	135	135
q7	1831	1776	1671	1671
q8	2429	2478	2499	2478
q9	7573	7338	7561	7338
q10	2863	3139	2692	2692
q11	568	471	482	471
q12	701	775	636	636
q13	3918	4407	3579	3579
q14	385	343	456	343
q15	953	805	798	798
q16	667	705	668	668
q17	1132	1347	1398	1347
q18	8105	7787	7979	7787
q19	870	852	851	851
q20	2055	2183	2000	2000
q21	4709	4618	4464	4464
q22	611	602	564	564
Total cold run time: 51916 ms
Total hot run time: 50008 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.25 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3742b8f748100bcdfd53efb48c68de63b7b98863, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.04	0.04
query3	0.25	0.09	0.08
query4	1.61	0.11	0.11
query5	0.27	0.25	0.26
query6	1.17	0.67	0.66
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.56	0.50	0.49
query10	0.55	0.54	0.55
query11	0.14	0.10	0.10
query12	0.14	0.10	0.11
query13	0.62	0.61	0.62
query14	1.07	1.05	1.05
query15	0.87	0.86	0.87
query16	0.38	0.40	0.40
query17	1.13	1.16	1.08
query18	0.23	0.22	0.21
query19	2.10	1.91	1.97
query20	0.02	0.01	0.02
query21	15.40	0.27	0.15
query22	4.96	0.06	0.05
query23	15.75	0.28	0.10
query24	0.91	0.67	0.33
query25	0.09	0.09	0.06
query26	0.15	0.15	0.14
query27	0.08	0.08	0.06
query28	4.53	1.14	0.97
query29	12.59	3.90	3.17
query30	0.28	0.13	0.11
query31	2.82	0.64	0.41
query32	3.25	0.59	0.49
query33	3.26	3.28	3.37
query34	16.47	5.38	4.72
query35	4.79	4.83	4.74
query36	0.66	0.51	0.48
query37	0.11	0.07	0.07
query38	0.08	0.04	0.04
query39	0.04	0.02	0.03
query40	0.20	0.16	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 97.93 s
Total hot run time: 28.25 s

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 9.46% (7/74) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants