Skip to content

[enhance](agg) Optimize AGG shuffle key selection#60572

Open
feiniaofeiafei wants to merge 8 commits intoapache:masterfrom
feiniaofeiafei:agg_shuffle_key_opt
Open

[enhance](agg) Optimize AGG shuffle key selection#60572
feiniaofeiafei wants to merge 8 commits intoapache:masterfrom
feiniaofeiafei:agg_shuffle_key_opt

Conversation

@feiniaofeiafei
Copy link
Contributor

@feiniaofeiafei feiniaofeiafei commented Feb 6, 2026

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@feiniaofeiafei
[enhance](agg)choose one agg gby key shuffle (apache#59449)
d128723 
This pr add variable choose_one_agg_shuffle_key, default value if false,
"set choose_one_agg_shuffle_key=true;" can make agg choose 1 key with
highest ndv to shuffle.

add

change visitPhysicalHashJoin in RequestPropertyDeriver

add in requestPropertyDeriver

add choose one partition key when specify partitionExpressions

unify choosing

support when parent request hash

add

fix

fix

remove 2+2 special logic

fix order of partitionExprs that lead to physicalProperties(GATHER) doesn't exist in root group

add test
@feiniaofeiafei
fix
049341a
@feiniaofeiafei
[enhance](memo)improve memo print
a02870a
@feiniaofeiafei
add test
48616bc
@feiniaofeiafei
add test
9c16307
@feiniaofeiafei
add join agg test
548c189
@Thearas
Copy link
Contributor

Thearas commented Feb 6, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Feb 6, 2026

run buildall

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Feb 6, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Feb 6, 2026

TPC-H: Total hot run time: 30094 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eba55ba6f5553983b079e9446a152fac35a36931, data reload: false

------ Round 1 ----------------------------------
q1	17645	4480	4267	4267
q2	2105	340	239	239
q3	10127	1274	717	717
q4	10195	771	309	309
q5	7537	2140	1934	1934
q6	197	182	150	150
q7	890	719	616	616
q8	9275	1370	1078	1078
q9	4684	4657	4585	4585
q10	6753	1939	1563	1563
q11	506	305	298	298
q12	332	370	223	223
q13	17761	4046	3225	3225
q14	250	237	218	218
q15	904	806	805	805
q16	703	689	628	628
q17	685	820	529	529
q18	6513	5933	5642	5642
q19	1212	1005	623	623
q20	495	492	392	392
q21	2500	1827	1774	1774
q22	362	326	279	279
Total cold run time: 101631 ms
Total hot run time: 30094 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4330	4296	4342	4296
q2	265	350	248	248
q3	2065	2651	2213	2213
q4	1327	1715	1304	1304
q5	4227	4243	4239	4239
q6	216	179	135	135
q7	1862	1797	1683	1683
q8	2504	2744	2476	2476
q9	7815	7794	7592	7592
q10	2964	3061	2611	2611
q11	562	498	461	461
q12	666	735	611	611
q13	3807	4407	3553	3553
q14	301	302	293	293
q15	843	815	785	785
q16	670	721	662	662
q17	1138	1368	1344	1344
q18	8397	8221	7798	7798
q19	868	900	883	883
q20	2032	2140	2040	2040
q21	4735	4516	4383	4383
q22	559	554	501	501
Total cold run time: 52153 ms
Total hot run time: 50111 ms

@doris-robot
Copy link

doris-robot commented Feb 6, 2026

ClickBench: Total hot run time: 28.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eba55ba6f5553983b079e9446a152fac35a36931, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.04	0.04
query3	0.27	0.09	0.08
query4	1.61	0.11	0.11
query5	0.26	0.26	0.25
query6	1.16	0.67	0.68
query7	0.04	0.03	0.02
query8	0.06	0.04	0.04
query9	0.57	0.50	0.50
query10	0.55	0.54	0.56
query11	0.14	0.10	0.09
query12	0.14	0.10	0.10
query13	0.64	0.61	0.62
query14	1.07	1.04	1.06
query15	0.88	0.88	0.86
query16	0.42	0.38	0.38
query17	1.10	1.18	1.13
query18	0.22	0.22	0.21
query19	2.13	2.00	1.96
query20	0.02	0.02	0.02
query21	15.42	0.26	0.14
query22	5.40	0.06	0.06
query23	16.13	0.29	0.11
query24	0.91	0.35	1.04
query25	0.09	0.15	0.14
query26	0.15	0.13	0.14
query27	0.08	0.07	0.06
query28	4.97	1.14	0.97
query29	12.62	3.89	3.12
query30	0.27	0.14	0.11
query31	2.82	0.65	0.40
query32	3.23	0.60	0.50
query33	3.22	3.26	3.28
query34	16.14	5.41	4.75
query35	4.89	4.88	4.84
query36	0.66	0.50	0.49
query37	0.11	0.07	0.07
query38	0.07	0.05	0.04
query39	0.05	0.03	0.03
query40	0.21	0.17	0.15
query41	0.09	0.03	0.03
query42	0.05	0.03	0.04
query43	0.04	0.04	0.04
Total cold run time: 99.05 s
Total hot run time: 28.52 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 6, 2026

FE UT Coverage Report

Increment line coverage 40.19% (168/418) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 6, 2026

FE Regression Coverage Report

Increment line coverage 79.67% (333/418) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@feiniaofeiafei @Thearas @doris-robot @hello-stephen