feat[array]: validate array encodings on write

[joseph-isaacs]

based on #6213

[cloudflare-workers-and-pages]

Deploying vortex-bench with    Cloudflare Pages

Latest commit:	32c5fa6
Status:	🚫  Deploy failed.

View logs

[codspeed-hq]

Merging this PR will degrade performance by 32.42%

❌ 1 regressed benchmark
✅ 1137 untouched benchmarks
⏩ 1384 skipped benchmarks¹

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	chunked_bool_into_canonical[(1000, 10)]	43 µs	63.7 µs	-32.42%

---

_{Comparing ji/add-write-filter (32c5fa6) with develop (76f19cb)}

Footnotes

1. 1384 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[github-actions]

Benchmarks: Random Access

Summary

• Overall: 1.005x ➖
• Vortex: 1.009x ➖
• Best: No improvements
• Worst: random-access/vortex-tokio-local-disk (1.009x)
• Significant (>10%): 0↑ 0↓

Detailed Results Table

name	PR 32c5fa6	base 2c198c4	ratio (PR/base)	unit	remark
random-access/parquet-tokio-local-disk	185689029	1.85663e+08	1.00014	ns
random-access/vortex-tokio-local-disk	1359535	1.34709e+06	1.00924	ns

[github-actions]

Benchmarks: TPC-H SF=1 on NVME

Summary

• Overall: 0.991x ➖
• Vortex: 0.990x ➖
• duckdb:vortex: 0.994x ➖
• datafusion:vortex: 0.987x ➖
• Best: tpch_q16/duckdb:vortex-compact (0.932x)
• Worst: tpch_q19/duckdb:vortex-file-compressed (1.105x)
• Significant (>10%): 0↑ 1↓

Detailed Results Table

name	PR 32c5fa6	base 2c198c4	ratio (PR/base)	unit	remark
tpch_q01/datafusion:arrow	67529004	6.8159e+07	0.990757	ns
tpch_q02/datafusion:arrow	23634952	2.29921e+07	1.02796	ns
tpch_q03/datafusion:arrow	35575856	3.60815e+07	0.985986	ns
tpch_q04/datafusion:arrow	28624007	2.92717e+07	0.977872	ns
tpch_q05/datafusion:arrow	91248811	9.24014e+07	0.987526	ns
tpch_q06/datafusion:arrow	20422389	2.06475e+07	0.989096	ns
tpch_q07/datafusion:arrow	167056021	1.73141e+08	0.964855	ns
tpch_q08/datafusion:arrow	45155617	4.4734e+07	1.00942	ns
tpch_q09/datafusion:arrow	74888985	7.35073e+07	1.0188	ns
tpch_q10/datafusion:arrow	56011941	5.52481e+07	1.01383	ns
tpch_q11/datafusion:arrow	11636411	1.14113e+07	1.01973	ns
tpch_q12/datafusion:arrow	56461069	5.65245e+07	0.998878	ns
tpch_q13/datafusion:arrow	49084751	5.59692e+07	0.876996	ns	🚀
tpch_q14/datafusion:arrow	24618818	2.68956e+07	0.915347	ns
tpch_q15/datafusion:arrow	47735924	5.31166e+07	0.898701	ns	🚀
tpch_q16/datafusion:arrow	23115249	2.44674e+07	0.944738	ns
tpch_q17/datafusion:arrow	83479957	7.51428e+07	1.11095	ns	🚨
tpch_q18/datafusion:arrow	158443623	1.60424e+08	0.987658	ns
tpch_q19/datafusion:arrow	38430758	3.91598e+07	0.981382	ns
tpch_q20/datafusion:arrow	39750652	3.86301e+07	1.02901	ns
tpch_q21/datafusion:arrow	217588130	2.30341e+08	0.944634	ns
tpch_q22/datafusion:arrow	21126419	2.15837e+07	0.978816	ns
tpch_q01/datafusion:parquet	133239723	1.11689e+08	1.19295	ns	🚨
tpch_q02/datafusion:parquet	81443239	8.18589e+07	0.994922	ns
tpch_q03/datafusion:parquet	75443550	8.79599e+07	0.857704	ns	🚀
tpch_q04/datafusion:parquet	55369502	5.65896e+07	0.97844	ns
tpch_q05/datafusion:parquet	170002983	1.72633e+08	0.984764	ns
tpch_q06/datafusion:parquet	46530752	3.64793e+07	1.27554	ns	🚨
tpch_q07/datafusion:parquet	187668236	1.87949e+08	0.998504	ns
tpch_q08/datafusion:parquet	168768722	1.68284e+08	1.00288	ns
tpch_q09/datafusion:parquet	223972148	2.17805e+08	1.02832	ns
tpch_q10/datafusion:parquet	193276119	1.93945e+08	0.996551	ns
tpch_q11/datafusion:parquet	59566995	6.23215e+07	0.955801	ns
tpch_q12/datafusion:parquet	127624015	1.23923e+08	1.02986	ns
tpch_q13/datafusion:parquet	198268184	2.06898e+08	0.95829	ns
tpch_q14/datafusion:parquet	87828210	8.66565e+07	1.01352	ns
tpch_q15/datafusion:parquet	72445282	6.94042e+07	1.04382	ns
tpch_q16/datafusion:parquet	53315976	5.41386e+07	0.984806	ns
tpch_q17/datafusion:parquet	177658762	1.80293e+08	0.985388	ns
tpch_q18/datafusion:parquet	231967735	2.41938e+08	0.958788	ns
tpch_q19/datafusion:parquet	92250851	8.92433e+07	1.0337	ns
tpch_q20/datafusion:parquet	129653570	1.28518e+08	1.00884	ns
tpch_q21/datafusion:parquet	180446623	1.93344e+08	0.933292	ns
tpch_q22/datafusion:parquet	39484329	4.16167e+07	0.948762	ns
tpch_q01/datafusion:vortex-file-compressed	53975482	5.45605e+07	0.989277	ns
tpch_q02/datafusion:vortex-file-compressed	57866941	5.87006e+07	0.985799	ns
tpch_q03/datafusion:vortex-file-compressed	30267434	2.95497e+07	1.02429	ns
tpch_q04/datafusion:vortex-file-compressed	20188058	2.0858e+07	0.967881	ns
tpch_q05/datafusion:vortex-file-compressed	81349915	8.23661e+07	0.987663	ns
tpch_q06/datafusion:vortex-file-compressed	11316836	1.11122e+07	1.01842	ns
tpch_q07/datafusion:vortex-file-compressed	94638907	9.33583e+07	1.01372	ns
tpch_q08/datafusion:vortex-file-compressed	70726782	7.28457e+07	0.970913	ns
tpch_q09/datafusion:vortex-file-compressed	88724565	9.3862e+07	0.945266	ns
tpch_q10/datafusion:vortex-file-compressed	82702730	8.49929e+07	0.973054	ns
tpch_q11/datafusion:vortex-file-compressed	21742748	2.25101e+07	0.965911	ns
tpch_q12/datafusion:vortex-file-compressed	37252482	3.68094e+07	1.01204	ns
tpch_q13/datafusion:vortex-file-compressed	32136531	3.32153e+07	0.967522	ns
tpch_q14/datafusion:vortex-file-compressed	29233168	3.01939e+07	0.968182	ns
tpch_q15/datafusion:vortex-file-compressed	27507741	2.85079e+07	0.964916	ns
tpch_q16/datafusion:vortex-file-compressed	37154377	3.89469e+07	0.953976	ns
tpch_q17/datafusion:vortex-file-compressed	76364461	8.01903e+07	0.952291	ns
tpch_q18/datafusion:vortex-file-compressed	107436220	1.0701e+08	1.00398	ns
tpch_q19/datafusion:vortex-file-compressed	33793125	3.42503e+07	0.986653	ns
tpch_q20/datafusion:vortex-file-compressed	53585808	5.24157e+07	1.02232	ns
tpch_q21/datafusion:vortex-file-compressed	102849320	1.0026e+08	1.02583	ns
tpch_q22/datafusion:vortex-file-compressed	18020739	1.77218e+07	1.01687	ns
tpch_q01/datafusion:vortex-compact	61543111	6.25876e+07	0.983312	ns
tpch_q02/datafusion:vortex-compact	59524655	5.86516e+07	1.01488	ns
tpch_q03/datafusion:vortex-compact	31478409	3.27845e+07	0.960162	ns
tpch_q04/datafusion:vortex-compact	24114452	2.35508e+07	1.02393	ns
tpch_q05/datafusion:vortex-compact	84868439	8.6947e+07	0.976093	ns
tpch_q06/datafusion:vortex-compact	18092884	1.73434e+07	1.04322	ns
tpch_q07/datafusion:vortex-compact	98851541	9.75123e+07	1.01373	ns
tpch_q08/datafusion:vortex-compact	75257974	7.57703e+07	0.993238	ns
tpch_q09/datafusion:vortex-compact	97831694	9.80389e+07	0.997886	ns
tpch_q10/datafusion:vortex-compact	86613151	8.95064e+07	0.967676	ns
tpch_q11/datafusion:vortex-compact	22929535	2.44162e+07	0.939112	ns
tpch_q12/datafusion:vortex-compact	45833107	4.65169e+07	0.9853	ns
tpch_q13/datafusion:vortex-compact	38045021	4.0122e+07	0.948234	ns
tpch_q14/datafusion:vortex-compact	34682003	3.54613e+07	0.978025	ns
tpch_q15/datafusion:vortex-compact	36851616	3.69107e+07	0.998399	ns
tpch_q16/datafusion:vortex-compact	39841721	4.08634e+07	0.974998	ns
tpch_q17/datafusion:vortex-compact	81144564	8.10787e+07	1.00081	ns
tpch_q18/datafusion:vortex-compact	109803784	1.12578e+08	0.975353	ns
tpch_q19/datafusion:vortex-compact	49129993	5.0453e+07	0.973777	ns
tpch_q20/datafusion:vortex-compact	58304410	6.02212e+07	0.968171	ns
tpch_q21/datafusion:vortex-compact	104851253	1.06223e+08	0.987088	ns
tpch_q22/datafusion:vortex-compact	16016200	1.58698e+07	1.00923	ns
tpch_q01/duckdb:parquet	80222068	7.92938e+07	1.01171	ns
tpch_q02/duckdb:parquet	48128704	4.74573e+07	1.01415	ns
tpch_q03/duckdb:parquet	83630496	7.83841e+07	1.06693	ns
tpch_q04/duckdb:parquet	60384345	6.09384e+07	0.990908	ns
tpch_q05/duckdb:parquet	74761490	8.61069e+07	0.86824	ns	🚀
tpch_q06/duckdb:parquet	22579376	2.29594e+07	0.983449	ns
tpch_q07/duckdb:parquet	93313606	8.26239e+07	1.12938	ns	🚨
tpch_q08/duckdb:parquet	99154714	9.88424e+07	1.00316	ns
tpch_q09/duckdb:parquet	147255138	1.7761e+08	0.829093	ns	🚀
tpch_q10/duckdb:parquet	150891070	1.39078e+08	1.08493	ns
tpch_q11/duckdb:parquet	25677636	2.64624e+07	0.970346	ns
tpch_q12/duckdb:parquet	59489893	7.32839e+07	0.811773	ns	🚀
tpch_q13/duckdb:parquet	271560922	2.75799e+08	0.984632	ns
tpch_q14/duckdb:parquet	64061923	5.68909e+07	1.12605	ns	🚨
tpch_q15/duckdb:parquet	28906543	2.88956e+07	1.00038	ns
tpch_q16/duckdb:parquet	59807926	5.97482e+07	1.001	ns
tpch_q17/duckdb:parquet	75492993	6.61828e+07	1.14067	ns	🚨
tpch_q18/duckdb:parquet	130450399	1.33411e+08	0.977805	ns
tpch_q19/duckdb:parquet	71260645	8.72135e+07	0.817083	ns	🚀
tpch_q20/duckdb:parquet	70182103	8.23131e+07	0.852624	ns	🚀
tpch_q21/duckdb:parquet	191835124	1.95184e+08	0.982844	ns
tpch_q22/duckdb:parquet	56335659	5.66685e+07	0.994127	ns
tpch_q01/duckdb:vortex-file-compressed	33348762	3.40938e+07	0.978146	ns
tpch_q02/duckdb:vortex-file-compressed	29348393	3.10713e+07	0.944551	ns
tpch_q03/duckdb:vortex-file-compressed	32489917	3.24822e+07	1.00024	ns
tpch_q04/duckdb:vortex-file-compressed	34754615	3.52753e+07	0.985239	ns
tpch_q05/duckdb:vortex-file-compressed	37174714	3.8537e+07	0.964651	ns
tpch_q06/duckdb:vortex-file-compressed	9627362	9.50152e+06	1.01324	ns
tpch_q07/duckdb:vortex-file-compressed	39594078	3.9585e+07	1.00023	ns
tpch_q08/duckdb:vortex-file-compressed	44652265	4.53638e+07	0.984314	ns
tpch_q09/duckdb:vortex-file-compressed	79136163	7.73061e+07	1.02367	ns
tpch_q10/duckdb:vortex-file-compressed	36250980	3.68506e+07	0.983728	ns
tpch_q11/duckdb:vortex-file-compressed	16424628	1.63252e+07	1.00609	ns
tpch_q12/duckdb:vortex-file-compressed	21932546	2.13934e+07	1.0252	ns
tpch_q13/duckdb:vortex-file-compressed	38037249	3.77615e+07	1.0073	ns
tpch_q14/duckdb:vortex-file-compressed	24904015	2.48281e+07	1.00306	ns
tpch_q15/duckdb:vortex-file-compressed	18008865	1.83149e+07	0.983292	ns
tpch_q16/duckdb:vortex-file-compressed	32741475	3.18973e+07	1.02647	ns
tpch_q17/duckdb:vortex-file-compressed	25508893	2.627e+07	0.971028	ns
tpch_q18/duckdb:vortex-file-compressed	53473720	5.33397e+07	1.00251	ns
tpch_q19/duckdb:vortex-file-compressed	33866067	3.06587e+07	1.10462	ns	🚨
tpch_q20/duckdb:vortex-file-compressed	36461584	3.69077e+07	0.987913	ns
tpch_q21/duckdb:vortex-file-compressed	100969697	1.01554e+08	0.994244	ns
tpch_q22/duckdb:vortex-file-compressed	19030711	1.92034e+07	0.991006	ns
tpch_q01/duckdb:vortex-compact	38669498	3.90784e+07	0.989537	ns
tpch_q02/duckdb:vortex-compact	41937722	4.15315e+07	1.00978	ns
tpch_q03/duckdb:vortex-compact	36869030	3.80427e+07	0.969148	ns
tpch_q04/duckdb:vortex-compact	39784770	4.01957e+07	0.989776	ns
tpch_q05/duckdb:vortex-compact	44585927	4.52129e+07	0.986133	ns
tpch_q06/duckdb:vortex-compact	15938982	1.62635e+07	0.980048	ns
tpch_q07/duckdb:vortex-compact	47643292	4.79726e+07	0.993135	ns
tpch_q08/duckdb:vortex-compact	53375515	5.43009e+07	0.982957	ns
tpch_q09/duckdb:vortex-compact	90199416	9.0802e+07	0.993364	ns
tpch_q10/duckdb:vortex-compact	43049025	4.28705e+07	1.00416	ns
tpch_q11/duckdb:vortex-compact	21626918	2.29241e+07	0.943413	ns
tpch_q12/duckdb:vortex-compact	36021880	3.52585e+07	1.02165	ns
tpch_q13/duckdb:vortex-compact	43164522	4.43799e+07	0.972615	ns
tpch_q14/duckdb:vortex-compact	29938497	3.00201e+07	0.997281	ns
tpch_q15/duckdb:vortex-compact	20854461	2.13054e+07	0.978832	ns
tpch_q16/duckdb:vortex-compact	39367479	4.22243e+07	0.932342	ns
tpch_q17/duckdb:vortex-compact	29136806	2.90783e+07	1.00201	ns
tpch_q18/duckdb:vortex-compact	61229206	6.07471e+07	1.00794	ns
tpch_q19/duckdb:vortex-compact	36267273	3.65401e+07	0.992533	ns
tpch_q20/duckdb:vortex-compact	44932433	4.43978e+07	1.01204	ns
tpch_q21/duckdb:vortex-compact	110292732	1.10167e+08	1.00114	ns
tpch_q22/duckdb:vortex-compact	21185094	2.11159e+07	1.00328	ns
tpch_q01/duckdb:duckdb	17603529	1.74956e+07	1.00617	ns
tpch_q02/duckdb:duckdb	14309471	1.48347e+07	0.964597	ns
tpch_q03/duckdb:duckdb	21246529	2.16246e+07	0.982518	ns
tpch_q04/duckdb:duckdb	21371232	2.21044e+07	0.966831	ns
tpch_q05/duckdb:duckdb	22222857	2.23119e+07	0.996008	ns
tpch_q06/duckdb:duckdb	7364295	6.97011e+06	1.05655	ns
tpch_q07/duckdb:duckdb	26943042	2.63285e+07	1.02334	ns
tpch_q08/duckdb:duckdb	26704199	2.6743e+07	0.998549	ns
tpch_q09/duckdb:duckdb	60836977	6.13415e+07	0.991774	ns
tpch_q10/duckdb:duckdb	61327964	5.76628e+07	1.06356	ns
tpch_q11/duckdb:duckdb	7123641	7.08076e+06	1.00606	ns
tpch_q12/duckdb:duckdb	16979014	1.78325e+07	0.95214	ns
tpch_q13/duckdb:duckdb	39739213	3.99345e+07	0.99511	ns
tpch_q14/duckdb:duckdb	22662321	2.2351e+07	1.01393	ns
tpch_q15/duckdb:duckdb	14494159	1.40339e+07	1.0328	ns
tpch_q16/duckdb:duckdb	24165570	2.49983e+07	0.966687	ns
tpch_q17/duckdb:duckdb	18135216	1.80626e+07	1.00402	ns
tpch_q18/duckdb:duckdb	40105877	4.05457e+07	0.989153	ns
tpch_q19/duckdb:duckdb	32778340	3.18891e+07	1.02788	ns
tpch_q20/duckdb:duckdb	26932541	2.69282e+07	1.00016	ns
tpch_q21/duckdb:duckdb	62652595	6.23134e+07	1.00544	ns
tpch_q22/duckdb:duckdb	26568908	2.66637e+07	0.996444	ns

[github-actions]

Benchmarks: FineWeb NVMe

Summary

• Overall: 1.017x ➖
• Vortex: 1.023x ➖
• duckdb:vortex: 0.990x ➖
• datafusion:vortex: 1.057x ➖
• Best: fineweb_q03/duckdb:vortex-file-compressed (0.705x)
• Worst: fineweb_q03/datafusion:vortex-file-compressed (1.567x)
• Significant (>10%): 2↑ 4↓

Detailed Results Table

name	PR 32c5fa6	base 2c198c4	ratio (PR/base)	unit	remark
fineweb_q00/datafusion:parquet	6572183	6.08873e+06	1.0794	ns
fineweb_q01/datafusion:parquet	274175752	2.76999e+08	0.989808	ns
fineweb_q02/datafusion:parquet	271088377	2.71789e+08	0.997421	ns
fineweb_q03/datafusion:parquet	265840169	2.65723e+08	1.00044	ns
fineweb_q04/datafusion:parquet	283342693	2.83223e+08	1.00042	ns
fineweb_q05/datafusion:parquet	281833090	2.81902e+08	0.999755	ns
fineweb_q06/datafusion:parquet	275281431	2.71459e+08	1.01408	ns
fineweb_q07/datafusion:parquet	270061148	2.67272e+08	1.01044	ns
fineweb_q08/datafusion:parquet	265764268	2.63856e+08	1.00723	ns
fineweb_q00/datafusion:vortex-file-compressed	5280280	5.28729e+06	0.998674	ns
fineweb_q01/datafusion:vortex-file-compressed	28877349	3.28965e+07	0.877824	ns	🚀
fineweb_q02/datafusion:vortex-file-compressed	34541332	3.34405e+07	1.03292	ns
fineweb_q03/datafusion:vortex-file-compressed	115245654	7.35673e+07	1.56653	ns	🚨
fineweb_q04/datafusion:vortex-file-compressed	223444870	2.12185e+08	1.05307	ns
fineweb_q05/datafusion:vortex-file-compressed	201096687	1.9388e+08	1.03722	ns
fineweb_q06/datafusion:vortex-file-compressed	73423180	6.76712e+07	1.085	ns
fineweb_q07/datafusion:vortex-file-compressed	81649806	7.2708e+07	1.12298	ns	🚨
fineweb_q08/datafusion:vortex-file-compressed	30978713	3.0132e+07	1.0281	ns
fineweb_q00/datafusion:vortex-compact	5921733	5.68601e+06	1.04146	ns
fineweb_q01/datafusion:vortex-compact	170022494	1.72579e+08	0.985186	ns
fineweb_q02/datafusion:vortex-compact	206415484	1.75786e+08	1.17424	ns	🚨
fineweb_q03/datafusion:vortex-compact	1288902318	1.25683e+09	1.02552	ns
fineweb_q04/datafusion:vortex-compact	1490659035	1.4702e+09	1.01392	ns
fineweb_q05/datafusion:vortex-compact	1149674537	1.13887e+09	1.00949	ns
fineweb_q06/datafusion:vortex-compact	598512371	6.14209e+08	0.974444	ns
fineweb_q07/datafusion:vortex-compact	638060550	6.2554e+08	1.02002	ns
fineweb_q08/datafusion:vortex-compact	28246618	2.56192e+07	1.10256	ns	🚨
fineweb_q00/duckdb:parquet	33370821	3.31829e+07	1.00566	ns
fineweb_q01/duckdb:parquet	99677846	9.79984e+07	1.01714	ns
fineweb_q02/duckdb:parquet	108624844	1.07444e+08	1.01099	ns
fineweb_q03/duckdb:parquet	313847528	3.14052e+08	0.999349	ns
fineweb_q04/duckdb:parquet	438712847	4.36721e+08	1.00456	ns
fineweb_q05/duckdb:parquet	412426735	4.08985e+08	1.00842	ns
fineweb_q06/duckdb:parquet	206534798	2.06477e+08	1.00028	ns
fineweb_q07/duckdb:parquet	219495180	2.20002e+08	0.997699	ns
fineweb_q08/duckdb:parquet	64334608	6.66914e+07	0.964661	ns
fineweb_q00/duckdb:vortex-file-compressed	4186806	4.40296e+06	0.950906	ns
fineweb_q01/duckdb:vortex-file-compressed	25259177	2.51934e+07	1.00261	ns
fineweb_q02/duckdb:vortex-file-compressed	27440994	2.73918e+07	1.0018	ns
fineweb_q03/duckdb:vortex-file-compressed	118301231	1.6779e+08	0.705056	ns	🚀
fineweb_q04/duckdb:vortex-file-compressed	201436798	1.91227e+08	1.05339	ns
fineweb_q05/duckdb:vortex-file-compressed	188536269	1.78193e+08	1.05804	ns
fineweb_q06/duckdb:vortex-file-compressed	57907702	5.75963e+07	1.00541	ns
fineweb_q07/duckdb:vortex-file-compressed	60137294	6.186e+07	0.972151	ns
fineweb_q08/duckdb:vortex-file-compressed	9787542	9.17932e+06	1.06626	ns
fineweb_q00/duckdb:vortex-compact	5559368	5.51943e+06	1.00724	ns
fineweb_q01/duckdb:vortex-compact	91543813	9.29349e+07	0.985031	ns
fineweb_q02/duckdb:vortex-compact	107375094	1.02361e+08	1.04898	ns
fineweb_q03/duckdb:vortex-compact	826517322	8.29375e+08	0.996554	ns
fineweb_q04/duckdb:vortex-compact	873147440	8.6672e+08	1.00742	ns
fineweb_q05/duckdb:vortex-compact	778887549	7.75343e+08	1.00457	ns
fineweb_q06/duckdb:vortex-compact	446111446	4.40275e+08	1.01326	ns
fineweb_q07/duckdb:vortex-compact	463409458	4.65629e+08	0.995233	ns
fineweb_q08/duckdb:vortex-compact	9889028	9.75266e+06	1.01398	ns