fix: canonicalize VarBin/FSST with >2GB buffers

[a10y]

Fixes a potential bug where decoding a large FSST and VarBin arrays results in an invalid VarBinViewArray.

When you have a large buffer that is, currently we generate a new VBV with the single buffer plus some views built against it. There will be trouble if the buffer is > 2GiB though.

This PR splits out a separate build_views function that takes a max_buffer_len parameter and as it generates views, it splits (zero-copy) the underlying buffer into segments of no more than max_buffer_len.

[codspeed-hq]

Merging this PR will degrade performance by 29.75%

⚡ 2 improved benchmarks
❌ 7 regressed benchmarks
✅ 1245 untouched benchmarks
⏩ 1254 skipped benchmarks¹

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	canonical_into_nullable[(10000, 10, 0.0)]	529.8 µs	445.1 µs	+19.04%
❌	Simulation	canonical_into_non_nullable[(10000, 100, 0.0)]	1.9 ms	2.7 ms	-29.75%
⚡	Simulation	canonical_into_nullable[(10000, 100, 0.0)]	4.9 ms	4.1 ms	+19.84%
❌	Simulation	canonical_into_non_nullable[(10000, 100, 0.1)]	3.7 ms	4.5 ms	-18.15%
❌	Simulation	canonical_into_non_nullable[(10000, 100, 0.01)]	2.1 ms	2.9 ms	-27.39%
❌	Simulation	into_canonical_non_nullable[(10000, 100, 0.1)]	3.8 ms	4.6 ms	-17.55%
❌	Simulation	into_canonical_non_nullable[(10000, 100, 0.0)]	1.9 ms	2.7 ms	-29.31%
❌	Simulation	into_canonical_non_nullable[(10000, 100, 0.01)]	2.2 ms	3 ms	-26.7%
❌	Simulation	into_canonical_nullable[(10000, 100, 0.0)]	4.4 ms	5.2 ms	-15.41%

---

_{Comparing fsst-canonical (78cea9c) with develop (5483037)}

Footnotes

1. 1254 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[a10y]

This is actually very conservative, and doesn't account for the presence of inlined strings.

There isn't really a way to get around this without lots of data copying

[a10y]

(FWIW this is already a problem on develop as well, except there everything lives in one giant buffer which sometimes is too large for VarBinView)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.89%. Comparing base (4bbafe7) to head (14d4764).

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[AdamGS]

is this a Java heresy?

[a10y]

Yea, there's some ambiguity here with the spec and arrow-rs diverges from arrow-cpp

Arrow-rs uses u32 for all of these fields.

But, the spec states that all offsets should be treated as signed. That's how the Java implementation works, and also how the arrow-cpp implementation works https://github.com/apache/arrow/blob/7820f672edbbd516661740db9c355f2bc42bf602/cpp/src/arrow/util/binary_view_util.h#L52-L58

So just to be on the safe side I use i32 here instead of u32

[joseph-isaacs]

is this the test that takes a while to run?

[a10y]

no this one is very small b/c i artificially constrain the max buffer len

[a10y]

but now that we're hitting the same codepath for varbin/fsst it's ok