Skip to content

Conversation

@alamb
Copy link
Contributor

@alamb alamb commented Dec 16, 2025

Which issue does this PR close?

Rationale for this change

Upgrade to latest arrow version

I am making this PR early to test the arrow release with DataFusion

What changes are included in this PR?

  1. Update arrow
  2. Updates for API

Are these changes tested?

Yes by CI

Are there any user-facing changes?

No

@github-actions github-actions bot added sqllogictest SQL Logic Tests (.slt) common Related to common crate proto Related to proto crate functions Changes to functions implementation datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate labels Dec 16, 2025
@alamb alamb force-pushed the alamb/update_arrow_57.2.0 branch from 7ec33db to 24d3a15 Compare December 18, 2025 21:04
# Upstream arrow-rs issue: https://github.com/apache/arrow-rs/issues/8841
# This should succeed after we receive the fix
query error Arrow error: Compute error: Internal Error: Cannot cast BinaryView to BinaryArray of expected type
query I
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Jefffrey fixed them upstream ❤️

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this fix will also close #19290; quickly tested checking this PR branch out and trying the test in the issue and it seems to execute successfully

# coerce structs with different field orders,
# (note the *value*s are from column2 but the field name is 'xxx', as the coerced
# type takes the field name from the last argument (column3)
# should keep the same field values
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests now work thanks to @brancz in

drop table t;

query error DataFusion error: Optimizer rule 'simplify_expressions' failed[\s\S]*Arrow error: Cast error: Cannot cast string 'a' to value of Float64 type
statement ok
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests now work thanks to @brancz in

@Dandandan
Copy link
Contributor

run benchmarks

@Jefffrey
Copy link
Contributor

FYI @tobixdev can keep an eye on this PR as it should include the fix from arrow-rs needed to do #19085 -> can start it once this PR merges

@alamb
Copy link
Contributor Author

alamb commented Dec 23, 2025

run benchmarks

@alamb-ghbot

This comment was marked as outdated.

@alamb-ghbot
Copy link

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_57.2.0
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_update_arrow_57.2 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │  2767.72 ms │              2671.31 ms │     no change │
│ QQuery 1     │  1251.11 ms │              1188.18 ms │ +1.05x faster │
│ QQuery 2     │  2449.97 ms │              2294.54 ms │ +1.07x faster │
│ QQuery 3     │  1166.19 ms │              1150.24 ms │     no change │
│ QQuery 4     │  2274.51 ms │              2284.93 ms │     no change │
│ QQuery 5     │ 28696.11 ms │             28601.98 ms │     no change │
│ QQuery 6     │  3914.27 ms │              3971.77 ms │     no change │
│ QQuery 7     │  3651.74 ms │              3616.21 ms │     no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 46171.62ms │
│ Total Time (alamb_update_arrow_57.2)   │ 45779.17ms │
│ Average Time (HEAD)                    │  5771.45ms │
│ Average Time (alamb_update_arrow_57.2) │  5722.40ms │
│ Queries Faster                         │          2 │
│ Queries Slower                         │          0 │
│ Queries with No Change                 │          6 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_update_arrow_57.2 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.07 ms │                 2.32 ms │  1.12x slower │
│ QQuery 1     │    48.99 ms │                49.41 ms │     no change │
│ QQuery 2     │   137.78 ms │               132.68 ms │     no change │
│ QQuery 3     │   149.93 ms │               155.75 ms │     no change │
│ QQuery 4     │  1092.93 ms │              1100.46 ms │     no change │
│ QQuery 5     │  1509.24 ms │              1485.47 ms │     no change │
│ QQuery 6     │     2.02 ms │                 2.05 ms │     no change │
│ QQuery 7     │    54.26 ms │                55.34 ms │     no change │
│ QQuery 8     │  1441.32 ms │              1459.08 ms │     no change │
│ QQuery 9     │  1789.92 ms │              1858.92 ms │     no change │
│ QQuery 10    │   352.62 ms │               367.90 ms │     no change │
│ QQuery 11    │   404.84 ms │               420.42 ms │     no change │
│ QQuery 12    │  1362.12 ms │              1389.97 ms │     no change │
│ QQuery 13    │  2027.32 ms │              2022.58 ms │     no change │
│ QQuery 14    │  1227.95 ms │              1258.69 ms │     no change │
│ QQuery 15    │  1251.14 ms │              1253.71 ms │     no change │
│ QQuery 16    │  2668.86 ms │              2682.87 ms │     no change │
│ QQuery 17    │  2667.08 ms │              2677.97 ms │     no change │
│ QQuery 18    │  5631.29 ms │              4942.96 ms │ +1.14x faster │
│ QQuery 19    │   119.93 ms │               123.27 ms │     no change │
│ QQuery 20    │  1949.89 ms │              1838.56 ms │ +1.06x faster │
│ QQuery 21    │  2242.78 ms │              2173.30 ms │     no change │
│ QQuery 22    │  3898.87 ms │              3720.39 ms │     no change │
│ QQuery 23    │ 22288.53 ms │             12109.53 ms │ +1.84x faster │
│ QQuery 24    │   212.76 ms │               207.87 ms │     no change │
│ QQuery 25    │   480.56 ms │               479.09 ms │     no change │
│ QQuery 26    │   229.81 ms │               221.38 ms │     no change │
│ QQuery 27    │  2808.68 ms │              2702.34 ms │     no change │
│ QQuery 28    │ 24994.86 ms │             21730.94 ms │ +1.15x faster │
│ QQuery 29    │   988.56 ms │               987.75 ms │     no change │
│ QQuery 30    │  1366.47 ms │              1332.78 ms │     no change │
│ QQuery 31    │  1375.51 ms │              1304.00 ms │ +1.05x faster │
│ QQuery 32    │  5458.33 ms │              4682.87 ms │ +1.17x faster │
│ QQuery 33    │  6464.69 ms │              5781.71 ms │ +1.12x faster │
│ QQuery 34    │  6078.81 ms │              5871.69 ms │     no change │
│ QQuery 35    │  1963.68 ms │              1936.37 ms │     no change │
│ QQuery 36    │    68.30 ms │                67.42 ms │     no change │
│ QQuery 37    │    45.32 ms │                46.41 ms │     no change │
│ QQuery 38    │    68.03 ms │                68.16 ms │     no change │
│ QQuery 39    │   104.91 ms │               104.49 ms │     no change │
│ QQuery 40    │    28.10 ms │                27.21 ms │     no change │
│ QQuery 41    │    22.06 ms │                21.91 ms │     no change │
│ QQuery 42    │    20.37 ms │                19.70 ms │     no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 107101.47ms │
│ Total Time (alamb_update_arrow_57.2)   │  90877.71ms │
│ Average Time (HEAD)                    │   2490.73ms │
│ Average Time (alamb_update_arrow_57.2) │   2113.44ms │
│ Queries Faster                         │           7 │
│ Queries Slower                         │           1 │
│ Queries with No Change                 │          35 │
│ Queries with Failure                   │           0 │
└────────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ alamb_update_arrow_57.2 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 137.38 ms │               132.73 ms │     no change │
│ QQuery 2     │  29.66 ms │                30.00 ms │     no change │
│ QQuery 3     │  35.07 ms │                38.82 ms │  1.11x slower │
│ QQuery 4     │  29.02 ms │                28.79 ms │     no change │
│ QQuery 5     │  88.44 ms │                88.90 ms │     no change │
│ QQuery 6     │  19.93 ms │                19.63 ms │     no change │
│ QQuery 7     │ 235.82 ms │               227.84 ms │     no change │
│ QQuery 8     │  35.73 ms │                37.59 ms │  1.05x slower │
│ QQuery 9     │ 101.05 ms │               107.10 ms │  1.06x slower │
│ QQuery 10    │  63.69 ms │                64.63 ms │     no change │
│ QQuery 11    │  19.00 ms │                18.98 ms │     no change │
│ QQuery 12    │  51.72 ms │                52.51 ms │     no change │
│ QQuery 13    │  47.56 ms │                47.23 ms │     no change │
│ QQuery 14    │  13.90 ms │                13.72 ms │     no change │
│ QQuery 15    │  24.23 ms │                24.41 ms │     no change │
│ QQuery 16    │  25.06 ms │                24.36 ms │     no change │
│ QQuery 17    │ 150.67 ms │               150.91 ms │     no change │
│ QQuery 18    │ 282.61 ms │               271.37 ms │     no change │
│ QQuery 19    │  48.16 ms │                36.40 ms │ +1.32x faster │
│ QQuery 20    │  61.49 ms │                49.56 ms │ +1.24x faster │
│ QQuery 21    │ 319.20 ms │               336.62 ms │  1.05x slower │
│ QQuery 22    │  17.60 ms │                17.32 ms │     no change │
└──────────────┴───────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 1836.99ms │
│ Total Time (alamb_update_arrow_57.2)   │ 1819.42ms │
│ Average Time (HEAD)                    │   83.50ms │
│ Average Time (alamb_update_arrow_57.2) │   82.70ms │
│ Queries Faster                         │         2 │
│ Queries Slower                         │         4 │
│ Queries with No Change                 │        16 │
│ Queries with Failure                   │         0 │
└────────────────────────────────────────┴───────────┘

@alamb
Copy link
Contributor Author

alamb commented Dec 24, 2025

run benchmarks

@alamb-ghbot

This comment was marked as outdated.

@alamb-ghbot
Copy link

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_57.2.0
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_update_arrow_57.2 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │  2700.49 ms │              2692.25 ms │     no change │
│ QQuery 1     │  1263.34 ms │              1195.98 ms │ +1.06x faster │
│ QQuery 2     │  2480.36 ms │              2333.51 ms │ +1.06x faster │
│ QQuery 3     │  1162.40 ms │              1144.25 ms │     no change │
│ QQuery 4     │  2306.63 ms │              2258.38 ms │     no change │
│ QQuery 5     │ 28497.50 ms │             28794.82 ms │     no change │
│ QQuery 6     │  3967.24 ms │              3968.46 ms │     no change │
│ QQuery 7     │  3740.76 ms │              3744.01 ms │     no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 46118.73ms │
│ Total Time (alamb_update_arrow_57.2)   │ 46131.65ms │
│ Average Time (HEAD)                    │  5764.84ms │
│ Average Time (alamb_update_arrow_57.2) │  5766.46ms │
│ Queries Faster                         │          2 │
│ Queries Slower                         │          0 │
│ Queries with No Change                 │          6 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_update_arrow_57.2 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.38 ms │                 2.13 ms │ +1.11x faster │
│ QQuery 1     │    49.21 ms │                49.38 ms │     no change │
│ QQuery 2     │   133.40 ms │               135.12 ms │     no change │
│ QQuery 3     │   154.79 ms │               157.63 ms │     no change │
│ QQuery 4     │  1105.55 ms │              1125.54 ms │     no change │
│ QQuery 5     │  1516.54 ms │              1494.42 ms │     no change │
│ QQuery 6     │     2.06 ms │                 2.10 ms │     no change │
│ QQuery 7     │    56.58 ms │                56.37 ms │     no change │
│ QQuery 8     │  1485.93 ms │              1437.34 ms │     no change │
│ QQuery 9     │  1838.47 ms │              1822.70 ms │     no change │
│ QQuery 10    │   358.08 ms │               366.87 ms │     no change │
│ QQuery 11    │   409.45 ms │               413.04 ms │     no change │
│ QQuery 12    │  1387.36 ms │              1361.49 ms │     no change │
│ QQuery 13    │  2037.73 ms │              2041.79 ms │     no change │
│ QQuery 14    │  1293.00 ms │              1289.01 ms │     no change │
│ QQuery 15    │  1304.98 ms │              1285.26 ms │     no change │
│ QQuery 16    │  2745.69 ms │              2707.38 ms │     no change │
│ QQuery 17    │  2710.65 ms │              2664.66 ms │     no change │
│ QQuery 18    │  5585.94 ms │              5010.55 ms │ +1.11x faster │
│ QQuery 19    │   123.53 ms │               123.08 ms │     no change │
│ QQuery 20    │  1914.61 ms │              1877.57 ms │     no change │
│ QQuery 21    │  2197.97 ms │              2162.31 ms │     no change │
│ QQuery 22    │  3795.84 ms │              3722.56 ms │     no change │
│ QQuery 23    │ 16226.93 ms │             12140.04 ms │ +1.34x faster │
│ QQuery 24    │   216.07 ms │               213.70 ms │     no change │
│ QQuery 25    │   467.36 ms │               475.61 ms │     no change │
│ QQuery 26    │   229.42 ms │               233.52 ms │     no change │
│ QQuery 27    │  2766.30 ms │              2672.27 ms │     no change │
│ QQuery 28    │ 24764.37 ms │             21932.32 ms │ +1.13x faster │
│ QQuery 29    │   988.07 ms │               991.64 ms │     no change │
│ QQuery 30    │  1338.37 ms │              1318.09 ms │     no change │
│ QQuery 31    │  1380.64 ms │              1339.78 ms │     no change │
│ QQuery 32    │  5129.67 ms │              4956.29 ms │     no change │
│ QQuery 33    │  6197.64 ms │              5667.21 ms │ +1.09x faster │
│ QQuery 34    │  6276.32 ms │              5994.54 ms │     no change │
│ QQuery 35    │  1970.40 ms │              1892.81 ms │     no change │
│ QQuery 36    │    67.22 ms │                69.20 ms │     no change │
│ QQuery 37    │    45.04 ms │                45.99 ms │     no change │
│ QQuery 38    │    66.49 ms │                67.71 ms │     no change │
│ QQuery 39    │   106.13 ms │               100.96 ms │     no change │
│ QQuery 40    │    27.49 ms │                26.55 ms │     no change │
│ QQuery 41    │    23.85 ms │                23.47 ms │     no change │
│ QQuery 42    │    19.86 ms │                20.00 ms │     no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 100517.36ms │
│ Total Time (alamb_update_arrow_57.2)   │  91490.02ms │
│ Average Time (HEAD)                    │   2337.61ms │
│ Average Time (alamb_update_arrow_57.2) │   2127.67ms │
│ Queries Faster                         │           5 │
│ Queries Slower                         │           0 │
│ Queries with No Change                 │          38 │
│ Queries with Failure                   │           0 │
└────────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ alamb_update_arrow_57.2 ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1     │ 136.35 ms │               136.96 ms │    no change │
│ QQuery 2     │  28.61 ms │                29.09 ms │    no change │
│ QQuery 3     │  38.54 ms │                38.63 ms │    no change │
│ QQuery 4     │  28.57 ms │                28.40 ms │    no change │
│ QQuery 5     │  87.49 ms │                87.27 ms │    no change │
│ QQuery 6     │  19.85 ms │                19.76 ms │    no change │
│ QQuery 7     │ 237.65 ms │               239.63 ms │    no change │
│ QQuery 8     │  34.95 ms │                35.45 ms │    no change │
│ QQuery 9     │ 107.19 ms │               107.03 ms │    no change │
│ QQuery 10    │  63.60 ms │                62.76 ms │    no change │
│ QQuery 11    │  17.62 ms │                17.92 ms │    no change │
│ QQuery 12    │  52.18 ms │                51.66 ms │    no change │
│ QQuery 13    │  47.47 ms │                47.88 ms │    no change │
│ QQuery 14    │  14.18 ms │                13.66 ms │    no change │
│ QQuery 15    │  24.01 ms │                24.27 ms │    no change │
│ QQuery 16    │  24.90 ms │                25.16 ms │    no change │
│ QQuery 17    │ 152.13 ms │               154.29 ms │    no change │
│ QQuery 18    │ 284.74 ms │               279.15 ms │    no change │
│ QQuery 19    │  37.80 ms │                37.16 ms │    no change │
│ QQuery 20    │  49.15 ms │                52.53 ms │ 1.07x slower │
│ QQuery 21    │ 307.03 ms │               315.54 ms │    no change │
│ QQuery 22    │  17.40 ms │                17.60 ms │    no change │
└──────────────┴───────────┴─────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 1811.41ms │
│ Total Time (alamb_update_arrow_57.2)   │ 1821.81ms │
│ Average Time (HEAD)                    │   82.34ms │
│ Average Time (alamb_update_arrow_57.2) │   82.81ms │
│ Queries Faster                         │         0 │
│ Queries Slower                         │         1 │
│ Queries with No Change                 │        21 │
│ Queries with Failure                   │         0 │
└────────────────────────────────────────┴───────────┘

@alamb
Copy link
Contributor Author

alamb commented Jan 6, 2026

run benchmarks

@alamb alamb force-pushed the alamb/update_arrow_57.2.0 branch from ff98f2f to 4155056 Compare January 6, 2026 22:53
@alamb-ghbot
Copy link

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_57.2.0 (70e2841) to 166ef81 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

@alamb-ghbot
Copy link

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_57.2.0
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_update_arrow_57.2 ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0     │  2419.66 ms │              2435.88 ms │    no change │
│ QQuery 1     │   928.43 ms │               931.95 ms │    no change │
│ QQuery 2     │  1919.23 ms │              1852.60 ms │    no change │
│ QQuery 3     │  1176.41 ms │              1134.39 ms │    no change │
│ QQuery 4     │  2319.77 ms │              2343.65 ms │    no change │
│ QQuery 5     │ 28296.40 ms │             28333.13 ms │    no change │
│ QQuery 6     │  3812.68 ms │              4011.85 ms │ 1.05x slower │
│ QQuery 7     │  3909.11 ms │              3801.45 ms │    no change │
└──────────────┴─────────────┴─────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 44781.68ms │
│ Total Time (alamb_update_arrow_57.2)   │ 44844.90ms │
│ Average Time (HEAD)                    │  5597.71ms │
│ Average Time (alamb_update_arrow_57.2) │  5605.61ms │
│ Queries Faster                         │          0 │
│ Queries Slower                         │          1 │
│ Queries with No Change                 │          7 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_update_arrow_57.2 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     1.50 ms │                 1.65 ms │  1.10x slower │
│ QQuery 1     │    49.50 ms │                53.56 ms │  1.08x slower │
│ QQuery 2     │   134.56 ms │               137.00 ms │     no change │
│ QQuery 3     │   152.97 ms │               161.67 ms │  1.06x slower │
│ QQuery 4     │  1090.28 ms │              1202.82 ms │  1.10x slower │
│ QQuery 5     │  1388.56 ms │              1473.87 ms │  1.06x slower │
│ QQuery 6     │     1.46 ms │                 1.79 ms │  1.22x slower │
│ QQuery 7     │    53.23 ms │                59.79 ms │  1.12x slower │
│ QQuery 8     │  1441.71 ms │              1559.23 ms │  1.08x slower │
│ QQuery 9     │  1787.15 ms │              2104.78 ms │  1.18x slower │
│ QQuery 10    │   334.39 ms │               365.40 ms │  1.09x slower │
│ QQuery 11    │   386.11 ms │               424.23 ms │  1.10x slower │
│ QQuery 12    │  1263.44 ms │              1401.23 ms │  1.11x slower │
│ QQuery 13    │  1950.70 ms │              2079.84 ms │  1.07x slower │
│ QQuery 14    │  1228.14 ms │              1352.82 ms │  1.10x slower │
│ QQuery 15    │  1245.15 ms │              1339.71 ms │  1.08x slower │
│ QQuery 16    │  2540.26 ms │              2690.26 ms │  1.06x slower │
│ QQuery 17    │  2519.94 ms │              2661.98 ms │  1.06x slower │
│ QQuery 18    │  5735.86 ms │              5209.22 ms │ +1.10x faster │
│ QQuery 19    │   119.75 ms │               124.82 ms │     no change │
│ QQuery 20    │  1886.02 ms │              1904.50 ms │     no change │
│ QQuery 21    │  2207.11 ms │              2240.74 ms │     no change │
│ QQuery 22    │  3710.17 ms │              3890.36 ms │     no change │
│ QQuery 23    │ 17570.69 ms │             12478.67 ms │ +1.41x faster │
│ QQuery 24    │   213.61 ms │               223.01 ms │     no change │
│ QQuery 25    │   455.25 ms │               471.34 ms │     no change │
│ QQuery 26    │   208.03 ms │               218.49 ms │  1.05x slower │
│ QQuery 27    │  2818.97 ms │              2791.64 ms │     no change │
│ QQuery 28    │ 24790.93 ms │             22004.60 ms │ +1.13x faster │
│ QQuery 29    │   979.69 ms │               951.72 ms │     no change │
│ QQuery 30    │  1352.14 ms │              1321.24 ms │     no change │
│ QQuery 31    │  1345.29 ms │              1343.04 ms │     no change │
│ QQuery 32    │  5214.55 ms │              4666.46 ms │ +1.12x faster │
│ QQuery 33    │  5663.96 ms │              5429.21 ms │     no change │
│ QQuery 34    │  6339.20 ms │              5848.00 ms │ +1.08x faster │
│ QQuery 35    │  2102.60 ms │              1905.58 ms │ +1.10x faster │
│ QQuery 36    │    68.62 ms │                66.24 ms │     no change │
│ QQuery 37    │    45.75 ms │                47.01 ms │     no change │
│ QQuery 38    │    66.90 ms │                64.31 ms │     no change │
│ QQuery 39    │   108.48 ms │               103.79 ms │     no change │
│ QQuery 40    │    28.04 ms │                26.79 ms │     no change │
│ QQuery 41    │    23.76 ms │                22.50 ms │ +1.06x faster │
│ QQuery 42    │    20.97 ms │                19.84 ms │ +1.06x faster │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 100645.39ms │
│ Total Time (alamb_update_arrow_57.2)   │  92444.76ms │
│ Average Time (HEAD)                    │   2340.59ms │
│ Average Time (alamb_update_arrow_57.2) │   2149.88ms │
│ Queries Faster                         │           8 │
│ Queries Slower                         │          18 │
│ Queries with No Change                 │          17 │
│ Queries with Failure                   │           0 │
└────────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ alamb_update_arrow_57.2 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 150.55 ms │               119.27 ms │ +1.26x faster │
│ QQuery 2     │  31.78 ms │                26.97 ms │ +1.18x faster │
│ QQuery 3     │  39.93 ms │                37.16 ms │ +1.07x faster │
│ QQuery 4     │  29.91 ms │                28.85 ms │     no change │
│ QQuery 5     │  89.36 ms │                86.82 ms │     no change │
│ QQuery 6     │  20.29 ms │                19.77 ms │     no change │
│ QQuery 7     │ 248.61 ms │               225.85 ms │ +1.10x faster │
│ QQuery 8     │  34.57 ms │                33.26 ms │     no change │
│ QQuery 9     │ 112.16 ms │               104.82 ms │ +1.07x faster │
│ QQuery 10    │  65.13 ms │                63.24 ms │     no change │
│ QQuery 11    │  18.27 ms │                17.53 ms │     no change │
│ QQuery 12    │  51.40 ms │                51.26 ms │     no change │
│ QQuery 13    │  48.50 ms │                47.23 ms │     no change │
│ QQuery 14    │  14.33 ms │                13.11 ms │ +1.09x faster │
│ QQuery 15    │  25.41 ms │                24.14 ms │     no change │
│ QQuery 16    │  25.40 ms │                24.06 ms │ +1.06x faster │
│ QQuery 17    │ 166.39 ms │               157.40 ms │ +1.06x faster │
│ QQuery 18    │ 296.26 ms │               274.99 ms │ +1.08x faster │
│ QQuery 19    │  39.33 ms │                37.84 ms │     no change │
│ QQuery 20    │  50.56 ms │                50.52 ms │     no change │
│ QQuery 21    │ 354.49 ms │               322.95 ms │ +1.10x faster │
│ QQuery 22    │  17.62 ms │                17.69 ms │     no change │
└──────────────┴───────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 1930.25ms │
│ Total Time (alamb_update_arrow_57.2)   │ 1784.72ms │
│ Average Time (HEAD)                    │   87.74ms │
│ Average Time (alamb_update_arrow_57.2) │   81.12ms │
│ Queries Faster                         │        10 │
│ Queries Slower                         │         0 │
│ Queries with No Change                 │        12 │
│ Queries with Failure                   │         0 │
└────────────────────────────────────────┴───────────┘

@alamb
Copy link
Contributor Author

alamb commented Jan 7, 2026

run benchmark clickbench_partitioned

@alamb-ghbot
Copy link

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_57.2.0 (70e2841) to 166ef81 diff using: clickbench_partitioned
Results will be posted here when complete

@alamb-ghbot
Copy link

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_57.2.0
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_update_arrow_57.2 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     1.42 ms │                 1.49 ms │     no change │
│ QQuery 1     │    48.59 ms │                49.83 ms │     no change │
│ QQuery 2     │   131.87 ms │               136.30 ms │     no change │
│ QQuery 3     │   152.53 ms │               155.23 ms │     no change │
│ QQuery 4     │  1119.89 ms │              1075.89 ms │     no change │
│ QQuery 5     │  1386.03 ms │              1356.49 ms │     no change │
│ QQuery 6     │     1.43 ms │                 1.49 ms │     no change │
│ QQuery 7     │    53.49 ms │                54.89 ms │     no change │
│ QQuery 8     │  1495.30 ms │              1429.83 ms │     no change │
│ QQuery 9     │  1922.02 ms │              1811.82 ms │ +1.06x faster │
│ QQuery 10    │   340.60 ms │               341.14 ms │     no change │
│ QQuery 11    │   397.07 ms │               398.81 ms │     no change │
│ QQuery 12    │  1351.61 ms │              1281.04 ms │ +1.06x faster │
│ QQuery 13    │  1941.56 ms │              1949.01 ms │     no change │
│ QQuery 14    │  1283.23 ms │              1241.92 ms │     no change │
│ QQuery 15    │  1312.69 ms │              1227.68 ms │ +1.07x faster │
│ QQuery 16    │  2630.84 ms │              2551.15 ms │     no change │
│ QQuery 17    │  2569.18 ms │              2537.88 ms │     no change │
│ QQuery 18    │  5925.70 ms │              4884.09 ms │ +1.21x faster │
│ QQuery 19    │   122.98 ms │               117.89 ms │     no change │
│ QQuery 20    │  1899.88 ms │              1856.44 ms │     no change │
│ QQuery 21    │  2197.16 ms │              2159.12 ms │     no change │
│ QQuery 22    │  3828.64 ms │              3753.56 ms │     no change │
│ QQuery 23    │ 19419.02 ms │             12248.86 ms │ +1.59x faster │
│ QQuery 24    │   211.82 ms │               215.52 ms │     no change │
│ QQuery 25    │   455.66 ms │               464.44 ms │     no change │
│ QQuery 26    │   217.53 ms │               212.44 ms │     no change │
│ QQuery 27    │  2703.36 ms │              2722.79 ms │     no change │
│ QQuery 28    │ 25047.07 ms │             21628.01 ms │ +1.16x faster │
│ QQuery 29    │   958.12 ms │               957.97 ms │     no change │
│ QQuery 30    │  1350.46 ms │              1274.65 ms │ +1.06x faster │
│ QQuery 31    │  1376.47 ms │              1323.87 ms │     no change │
│ QQuery 32    │  5140.76 ms │              4903.25 ms │     no change │
│ QQuery 33    │  5611.26 ms │              5551.41 ms │     no change │
│ QQuery 34    │  5979.56 ms │              5541.44 ms │ +1.08x faster │
│ QQuery 35    │  1986.36 ms │              1884.19 ms │ +1.05x faster │
│ QQuery 36    │    66.47 ms │                69.69 ms │     no change │
│ QQuery 37    │    45.08 ms │                45.64 ms │     no change │
│ QQuery 38    │    64.65 ms │                66.74 ms │     no change │
│ QQuery 39    │   101.95 ms │               104.09 ms │     no change │
│ QQuery 40    │    27.10 ms │                26.46 ms │     no change │
│ QQuery 41    │    22.14 ms │                23.62 ms │  1.07x slower │
│ QQuery 42    │    18.92 ms │                19.96 ms │  1.05x slower │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 102917.47ms │
│ Total Time (alamb_update_arrow_57.2)   │  89658.02ms │
│ Average Time (HEAD)                    │   2393.43ms │
│ Average Time (alamb_update_arrow_57.2) │   2085.07ms │
│ Queries Faster                         │           9 │
│ Queries Slower                         │           2 │
│ Queries with No Change                 │          32 │
│ Queries with Failure                   │           0 │
└────────────────────────────────────────┴─────────────┘

@alamb
Copy link
Contributor Author

alamb commented Jan 7, 2026

run benchmark clickbench_partitioned

@alamb-ghbot
Copy link

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_57.2.0 (70e2841) to 166ef81 diff using: clickbench_partitioned
Results will be posted here when complete

@alamb-ghbot
Copy link

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_57.2.0
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_update_arrow_57.2 ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     1.42 ms │                 1.46 ms │     no change │
│ QQuery 1     │    50.06 ms │                50.90 ms │     no change │
│ QQuery 2     │   131.12 ms │               134.76 ms │     no change │
│ QQuery 3     │   153.53 ms │               157.05 ms │     no change │
│ QQuery 4     │  1220.63 ms │              1017.31 ms │ +1.20x faster │
│ QQuery 5     │  1500.76 ms │              1342.79 ms │ +1.12x faster │
│ QQuery 6     │     1.46 ms │                 1.46 ms │     no change │
│ QQuery 7     │    52.94 ms │                54.49 ms │     no change │
│ QQuery 8     │  1559.99 ms │              1402.36 ms │ +1.11x faster │
│ QQuery 9     │  1924.97 ms │              1795.37 ms │ +1.07x faster │
│ QQuery 10    │   344.87 ms │               344.50 ms │     no change │
│ QQuery 11    │   396.88 ms │               386.58 ms │     no change │
│ QQuery 12    │  1375.56 ms │              1266.24 ms │ +1.09x faster │
│ QQuery 13    │  2086.16 ms │              1898.05 ms │ +1.10x faster │
│ QQuery 14    │  1313.50 ms │              1205.10 ms │ +1.09x faster │
│ QQuery 15    │  1325.84 ms │              1164.76 ms │ +1.14x faster │
│ QQuery 16    │  2620.77 ms │              2481.31 ms │ +1.06x faster │
│ QQuery 17    │  2569.29 ms │              2452.75 ms │     no change │
│ QQuery 18    │  4761.52 ms │              4751.51 ms │     no change │
│ QQuery 19    │   121.61 ms │               117.08 ms │     no change │
│ QQuery 20    │  1844.33 ms │              1887.06 ms │     no change │
│ QQuery 21    │  2144.52 ms │              2167.24 ms │     no change │
│ QQuery 22    │  3658.67 ms │              3714.86 ms │     no change │
│ QQuery 23    │ 12063.44 ms │             12123.87 ms │     no change │
│ QQuery 24    │   208.61 ms │               203.13 ms │     no change │
│ QQuery 25    │   464.58 ms │               459.26 ms │     no change │
│ QQuery 26    │   211.87 ms │               224.95 ms │  1.06x slower │
│ QQuery 27    │  2622.81 ms │              2729.67 ms │     no change │
│ QQuery 28    │ 24356.16 ms │             21846.79 ms │ +1.11x faster │
│ QQuery 29    │   981.66 ms │               964.46 ms │     no change │
│ QQuery 30    │  1338.13 ms │              1290.33 ms │     no change │
│ QQuery 31    │  1335.72 ms │              1306.09 ms │     no change │
│ QQuery 32    │  5035.68 ms │              4874.55 ms │     no change │
│ QQuery 33    │  5371.04 ms │              5322.62 ms │     no change │
│ QQuery 34    │  5231.42 ms │              5374.48 ms │     no change │
│ QQuery 35    │  1923.27 ms │              1869.69 ms │     no change │
│ QQuery 36    │    65.19 ms │                65.93 ms │     no change │
│ QQuery 37    │    45.02 ms │                45.54 ms │     no change │
│ QQuery 38    │    66.03 ms │                63.26 ms │     no change │
│ QQuery 39    │   102.85 ms │                99.56 ms │     no change │
│ QQuery 40    │    25.12 ms │                25.50 ms │     no change │
│ QQuery 41    │    22.95 ms │                22.36 ms │     no change │
│ QQuery 42    │    19.04 ms │                18.77 ms │     no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 92651.00ms │
│ Total Time (alamb_update_arrow_57.2)   │ 88725.81ms │
│ Average Time (HEAD)                    │  2154.67ms │
│ Average Time (alamb_update_arrow_57.2) │  2063.39ms │
│ Queries Faster                         │         10 │
│ Queries Slower                         │          1 │
│ Queries with No Change                 │         32 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

alamb added a commit to apache/arrow-rs that referenced this pull request Jan 7, 2026
# Which issue does this PR close?


- Part of #8465

# Rationale for this change

- See  #8465

# What changes are included in this PR?

1. Update version to 57.2.0
2. Add CHANGELOG. See rendered version
https://github.com/alamb/arrow-rs/blob/alamb/prepare_57.2.0/CHANGELOG.md

# Are these changes tested?
By CI and I am testing them manually in DataFusion here:
- apache/datafusion#19355

# Are there any user-facing changes?

Version and changelog
@alamb
Copy link
Contributor Author

alamb commented Jan 7, 2026

I do see a reproducable speedup on q28 which I think is real

│ QQuery 28    │ 24356.16 ms │             21846.79 ms │ +1.11x faster │

Query 28 is:

SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length("Referer")) AS l, COUNT(*) AS c, MIN("Referer") FROM hits WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;

SELECT REGEXP_REPLACE("Referer", '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length("Referer")) AS l, COUNT(*) AS c, MIN("Referer") FROM hits WHERE "Referer" <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25;

It is not clear to me what is responsible

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate datasource Changes to the datasource crate functions Changes to functions implementation physical-plan Changes to the physical-plan crate proto Related to proto crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants