Skip to content

Conversation

@coderfender
Copy link
Contributor

@coderfender coderfender commented Oct 17, 2025

Which issue does this PR close?

Closes #531 .
(Partially closes 531) . The ANSI changes to support AVG will be tracked in a new PR

Rationale for this change

DataFusion's default SUM doesn't match Spark's overflow semantics. This implementation ensures we get the exact same behavior as Spark for integer overflow in all 3 eval modes (ANSI , Legacy and Try mode)

What changes are included in this PR?

This PR adds native Rust implementation for SUM aggregation on integer types (byte, short, int, long)

Native changes (Rust):
(Inspired from SumDecimal and spark's SUM impl)

  1. New SumInteger aggregate function that handles SUM for all integer types (in coherence with Spark)
  2. Support all eval modes (ANSI , Legacy and Try)
    ( Implemented code in similar fashion of spark leveraging Option<i64> to represent NULL and numeric values for sum , and an additional parameter called has_all_nulls which is leveraged in Try mode to distinguish if NULL sum is caused by all NULL inputs or the fact that the sum overflowed. (Spark does this with shouldTrackIsEmpty and assigning NULL to running sum which is a long datatype) )
  3. Implemented groups accumulator (which should optimize group by op)

Scala side changes :

  1. Update CometSum to add ANSI support (ANSI and Try)
  2. Change proto schema to pass along eval_mode instead of fail_on_error to support Legacy, ANSI and Try eval modes
  3. Added tests for NULL (had to force cast nulls as java.lang.Long to avoid Scala's auto-boxing feature which auto casts objects to primitive types there by casting nulls to 0s) handling in both simple and GROUP BY agg .
  4. Added tests for both ANSI and non-ANSI mode tests for the input types Int, Short, Byte
  5. Added try_sum overflow tests with GROUP BY (mixed overflow/non-overflow groups)

How are these changes tested?

  1. Unit tests on Scala side to test various aggregation use cases in all 3 eval modes .
  2. Added new tests for groupBy use cases to test out groups accumulator

@coderfender coderfender marked this pull request as draft October 17, 2025 19:13
@coderfender
Copy link
Contributor Author

Draft PR to support sum function - WIP

@coderfender coderfender marked this pull request as ready for review October 28, 2025 07:53
@coderfender coderfender changed the title [WIP]: Support ANSI mode sum expr feat: Support ANSI mode sum expr Oct 29, 2025
@codecov-commenter
Copy link

codecov-commenter commented Oct 30, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.59%. Comparing base (f09f8af) to head (9effc42).
⚠️ Report is 797 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2600      +/-   ##
============================================
+ Coverage     56.12%   59.59%   +3.46%     
- Complexity      976     1376     +400     
============================================
  Files           119      167      +48     
  Lines         11743    15492    +3749     
  Branches       2251     2567     +316     
============================================
+ Hits           6591     9232    +2641     
- Misses         4012     4962     +950     
- Partials       1140     1298     +158     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderfender coderfender force-pushed the support_overflow_sum_function branch from 970d693 to c0715aa Compare November 7, 2025 01:16
@coderfender
Copy link
Contributor Author

@andygrove , @comphead I believe this should be ready for review (pending CI)

@comphead
Copy link
Contributor

comphead commented Nov 7, 2025

- Windows support *** FAILED *** (3 seconds, 973 milliseconds)
  Expected only Comet native operators, but found Project.
  plan: Project
  +- Window
     +- CometExchange
        +- CometScan [native_iceberg_compat] parquet

Thats actually interesting why would window fall back, we planning to fallback on windows in #2726 but the test would still preserve windows, so I think we need to check a fallback reason

@coderfender coderfender force-pushed the support_overflow_sum_function branch from 6dcf47b to 13b7d68 Compare November 11, 2025 22:02
@coderfender
Copy link
Contributor Author

- Windows support *** FAILED *** (3 seconds, 973 milliseconds)
  Expected only Comet native operators, but found Project.
  plan: Project
  +- Window
     +- CometExchange
        +- CometScan [native_iceberg_compat] parquet

Thats actually interesting why would window fall back, we planning to fallback on windows in #2726 but the test would still preserve windows, so I think we need to check a fallback reason

This is no longer a failing test after I rebased with main branch

@coderfender
Copy link
Contributor Author

@andygrove @comphead . All the tests passed and the PR is rest for review

Copy link
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderfender This LGTM. I am going to pull the PR locally and do some testing/benchmarking today. Could you rebease/upmerge?

@coderfender
Copy link
Contributor Author

Rebased with main

Copy link
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @coderfender. LGTM pending CI.

@andygrove
Copy link
Member

@comphead could you also review this PR when you have time?

@coderfender
Copy link
Contributor Author

Fixed clippy errors

@coderfender
Copy link
Contributor Author

coderfender commented Dec 19, 2025

Thank you for the approval @andygrove

"null_tbl") {
val res = sql("SELECT try_sum(_1) FROM null_tbl")
checkSparkAnswerAndOperator(res)
assert(res.collect() === Array(Row(null)))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we really need this assertion after checkSparkAnswerAndOperator ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@comphead , Yeah , I found some inconsistencies while writing these tests as (partially) discussed here : #2692 . Also ,I wanted to be completely sure in terms of output given the core functionality of this PR

Copy link
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @coderfender for this great PR, I left couple of nits and personally I'm feeling separate implementation would be easier to navigate and support, WDYT?

@coderfender
Copy link
Contributor Author

Thank you for the comments @comphead . I addressed the review comments and pushed a fix. Please take a look whenever you get a chance

@coderfender coderfender requested a review from comphead December 22, 2025 23:10
Copy link
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @coderfender this is an epic PR

@comphead comphead merged commit f232887 into apache:main Dec 23, 2025
118 checks passed
@coderfender
Copy link
Contributor Author

Thank you @andygrove , @comphead for the guidance and mentorship :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add ANSI support in SUM and AVG

5 participants