Skip to content

[feat](doris-catalog) doris cross-cluster insert#60557

Open
TsukiokaKogane wants to merge 7 commits intoapache:masterfrom
TsukiokaKogane:support_remote_catalog_insert_public
Open

[feat](doris-catalog) doris cross-cluster insert#60557
TsukiokaKogane wants to merge 7 commits intoapache:masterfrom
TsukiokaKogane:support_remote_catalog_insert_public

Conversation

@TsukiokaKogane
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #60295

Problem Summary:

  1. support insert & insert overwrite on remote doris catalog
  2. support temporary partitions on remote doris catalog
  3. fix remote doris catalog rpc timeout unit
  4. DO NOT support txn insert & group commit

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@TsukiokaKogane
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.29% (1792/2260)
Line Coverage 64.74% (31825/49158)
Region Coverage 65.41% (15882/24280)
Branch Coverage 55.94% (8435/15078)

@doris-robot
Copy link

TPC-H: Total hot run time: 31471 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97168897275f4683257ec006a83a0f0b792a824d, data reload: false

------ Round 1 ----------------------------------
q1	17660	4522	4407	4407
q2	2051	349	224	224
q3	10469	1342	767	767
q4	10343	809	329	329
q5	9507	2263	1927	1927
q6	228	182	156	156
q7	883	750	590	590
q8	9266	1447	1221	1221
q9	5356	4836	4890	4836
q10	6888	1959	1582	1582
q11	503	304	290	290
q12	398	377	231	231
q13	17813	4066	3274	3274
q14	231	246	214	214
q15	924	811	804	804
q16	689	684	621	621
q17	708	767	547	547
q18	6755	6412	6330	6330
q19	1326	1004	654	654
q20	395	357	256	256
q21	2709	2004	1937	1937
q22	358	318	274	274
Total cold run time: 105460 ms
Total hot run time: 31471 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4442	4333	4374	4333
q2	257	330	265	265
q3	2137	2601	2229	2229
q4	1372	1735	1327	1327
q5	4327	4207	4182	4182
q6	216	184	140	140
q7	1884	2156	1917	1917
q8	2654	2501	2431	2431
q9	7635	7459	7620	7459
q10	2784	3063	2750	2750
q11	570	503	455	455
q12	684	752	658	658
q13	3851	4494	3582	3582
q14	305	302	288	288
q15	851	802	793	793
q16	694	720	728	720
q17	1198	1489	1385	1385
q18	8328	8094	7920	7920
q19	936	920	914	914
q20	2119	2144	1962	1962
q21	4524	4352	4109	4109
q22	544	564	504	504
Total cold run time: 52312 ms
Total hot run time: 50323 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.23 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 97168897275f4683257ec006a83a0f0b792a824d, data reload: false

query1	0.05	0.05	0.05
query2	0.13	0.07	0.07
query3	0.32	0.07	0.07
query4	1.61	0.10	0.10
query5	0.25	0.25	0.24
query6	1.15	0.66	0.64
query7	0.03	0.02	0.02
query8	0.08	0.06	0.06
query9	0.59	0.50	0.49
query10	0.54	0.56	0.54
query11	0.27	0.13	0.14
query12	0.27	0.14	0.14
query13	0.64	0.63	0.60
query14	0.98	0.98	0.98
query15	0.94	0.83	0.83
query16	0.39	0.40	0.39
query17	1.06	1.04	1.05
query18	0.26	0.23	0.23
query19	1.84	1.83	1.79
query20	0.02	0.01	0.02
query21	15.38	0.34	0.29
query22	4.98	0.11	0.12
query23	15.36	0.45	0.28
query24	2.30	0.56	0.39
query25	0.11	0.11	0.11
query26	0.19	0.18	0.18
query27	0.10	0.10	0.10
query28	3.62	1.14	0.98
query29	12.52	4.12	3.31
query30	0.34	0.12	0.11
query31	2.79	0.70	0.44
query32	3.24	0.64	0.51
query33	2.95	3.11	3.04
query34	16.30	5.07	4.46
query35	4.47	4.47	4.48
query36	0.62	0.50	0.50
query37	0.30	0.10	0.09
query38	0.26	0.06	0.06
query39	0.08	0.05	0.04
query40	0.22	0.17	0.18
query41	0.13	0.08	0.06
query42	0.08	0.05	0.05
query43	0.06	0.06	0.06
Total cold run time: 97.82 s
Total hot run time: 28.23 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 13.33% (2/15) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.56% (19376/36862)
Line Coverage 36.06% (180041/499291)
Region Coverage 32.42% (139548/430471)
Branch Coverage 33.42% (60437/180846)

@TsukiokaKogane
Copy link
Contributor Author

run buildall

@hello-stephen
Copy link
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.29% (1792/2260)
Line Coverage 64.75% (31831/49158)
Region Coverage 65.42% (15884/24280)
Branch Coverage 55.97% (8439/15078)

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 2.44% (29/1189) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

TPC-H: Total hot run time: 31323 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f3938347c127a752b04e13d9f40db00f84820c1, data reload: false

------ Round 1 ----------------------------------
q1	17617	4508	4290	4290
q2	2054	355	238	238
q3	10153	1364	770	770
q4	10211	824	325	325
q5	7499	2248	1990	1990
q6	193	178	147	147
q7	909	751	600	600
q8	9262	1405	1161	1161
q9	4847	4686	4653	4653
q10	6857	1960	1548	1548
q11	538	322	304	304
q12	336	387	230	230
q13	17778	4095	3252	3252
q14	241	234	218	218
q15	881	817	798	798
q16	672	681	642	642
q17	708	861	520	520
q18	6624	6218	6365	6218
q19	1201	1083	688	688
q20	575	545	421	421
q21	2924	2019	2068	2019
q22	380	335	291	291
Total cold run time: 102460 ms
Total hot run time: 31323 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4618	4530	4678	4530
q2	264	342	272	272
q3	2466	2879	2389	2389
q4	1520	1937	1503	1503
q5	4636	4691	4697	4691
q6	228	186	139	139
q7	1953	1882	1745	1745
q8	2553	2367	2398	2367
q9	7691	7717	7420	7420
q10	2944	3124	2614	2614
q11	540	482	468	468
q12	675	730	617	617
q13	3818	4335	3613	3613
q14	290	296	269	269
q15	825	775	782	775
q16	661	690	636	636
q17	1096	1274	1285	1274
q18	7534	7362	7433	7362
q19	897	847	841	841
q20	1977	2104	1878	1878
q21	4597	4245	4078	4078
q22	560	544	506	506
Total cold run time: 52343 ms
Total hot run time: 49987 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.66 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9f3938347c127a752b04e13d9f40db00f84820c1, data reload: false

query1	0.05	0.05	0.05
query2	0.13	0.07	0.07
query3	0.32	0.08	0.08
query4	1.61	0.10	0.10
query5	0.28	0.25	0.25
query6	1.15	0.65	0.65
query7	0.03	0.03	0.02
query8	0.07	0.06	0.06
query9	0.58	0.51	0.50
query10	0.55	0.56	0.54
query11	0.27	0.14	0.14
query12	0.26	0.14	0.15
query13	0.62	0.60	0.60
query14	0.98	0.99	0.98
query15	0.89	0.82	0.83
query16	0.45	0.39	0.39
query17	0.99	1.08	0.97
query18	0.25	0.23	0.23
query19	1.94	1.85	1.88
query20	0.02	0.02	0.01
query21	15.40	0.34	0.28
query22	4.92	0.11	0.12
query23	15.34	0.44	0.27
query24	2.34	0.56	0.37
query25	0.11	0.10	0.11
query26	0.19	0.18	0.19
query27	0.11	0.11	0.10
query28	3.62	1.16	0.99
query29	12.54	4.13	3.34
query30	0.33	0.13	0.11
query31	2.79	0.70	0.45
query32	3.24	0.62	0.50
query33	3.00	3.06	3.01
query34	16.27	5.10	4.42
query35	4.53	4.91	5.05
query36	0.69	0.58	0.55
query37	0.33	0.08	0.09
query38	0.28	0.06	0.06
query39	0.08	0.05	0.05
query40	0.20	0.18	0.16
query41	0.14	0.08	0.07
query42	0.09	0.05	0.05
query43	0.06	0.06	0.05
Total cold run time: 98.04 s
Total hot run time: 28.66 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 13.33% (2/15) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.61% (19409/36895)
Line Coverage 36.12% (180587/499982)
Region Coverage 32.47% (140040/431269)
Branch Coverage 33.48% (60631/181101)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 73.33% (11/15) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.72% (25930/36156)
Line Coverage 54.39% (271270/498758)
Region Coverage 51.96% (226378/435653)
Branch Coverage 53.36% (97020/181809)

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 29.44% (350/1189) 🎉
Increment coverage report
Complete coverage report

@morningman morningman self-assigned this Feb 9, 2026
@TsukiokaKogane
Copy link
Contributor Author

run buildall

@morningman morningman changed the title [Feature] doris cross-cluster insert [feat](doris-catalog) doris cross-cluster insert Feb 9, 2026
@morningman
Copy link
Contributor

run buildall

// Remote transaction request and Result definitions for cross-cluster export.
// These structs are used by beginRemoteTxn/commitRemoteTxn/abortRemoteTxn RPCs.
struct TBeginRemoteTxnRequest {
1: required string user
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

always use optional for all fields

1: required string user
2: required string passwd
3: optional string token
4: required string db
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better add catalog, even if currently only support "internal" catalog

} catch (Exception e) {
long costMs = System.currentTimeMillis() - startTime;
LOG.warn("get master address for catalog {} failed, cost={}ms", name, costMs, e);
throw new RuntimeException(e.getMessage());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
throw new RuntimeException(e.getMessage());
throw new RuntimeException(Util.getRootCause(e), e);

Same suggestion for other places

contains("VOlapScanNode")
}
sql """
insert OVERWRITE table `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_insert` select /*+ SET_VAR(enable_nereids_distribute_planner=true, enable_sql_cache=true) */ * from `${catalog_name}`.`${db_name}`.`test_remote_doris_all_types_select_t`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the behavior of setting these variables?
Does it affect remote cluster?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And why we set these 2 variables?

@doris-robot
Copy link

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.35% (1795/2262)
Line Coverage 64.87% (31962/49269)
Region Coverage 65.53% (15940/24323)
Branch Coverage 56.09% (8478/15114)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants