perf: Initialize tokio runtime worker threads from spark.executor.cores

[andygrove]

Summary

• Initialize the tokio runtime worker thread count from spark.executor.cores instead of defaulting to all CPU cores
• Resolves executor cores on the Scala side via numDriverOrExecutorCores() (handles local, local[N], local[*], and cluster modes) and injects the resolved value into the native config map
• Replaces Lazy<Runtime> with OnceLock<Runtime> so the runtime can be initialized with the correct thread count from createPlan()

Priority chain (highest to lowest)

1. COMET_WORKER_THREADS env var — explicit override (unchanged)
2. spark.executor.cores — Spark-aware default
3. Tokio default (all CPU cores) — only if get_runtime() called before createPlan

Test plan

• cd native && cargo build passes
• Existing tests pass (runtime API is unchanged)
• Check logs for "Comet tokio runtime" message showing correct thread count
• Verify env var override: COMET_WORKER_THREADS=2 still takes precedence

🤖 Generated with Claude Code

[andygrove]

@sqlbenchmark run tpch

[sqlbenchmark]

Comet TPC-H Benchmark Results

Baseline: main (f697d27)
PR: d575428 - fmt
Scale Factor: SF100
Iterations: 1

Query Times

Query	Baseline Avg (s)	Baseline Best (s)	PR Avg (s)	PR Best (s)	Change (Avg)	Change (Best)
Q1	10.44	10.44	10.46	10.46	⚪ +0.2%	⚪ +0.2%
Q2	5.79	5.79	5.89	5.89	⚪ +1.6%	⚪ +1.6%
Q3	9.70	9.70	9.72	9.72	⚪ +0.2%	⚪ +0.2%
Q4	11.35	11.35	11.52	11.52	⚪ +1.5%	⚪ +1.5%
Q5	18.72	18.72	19.11	19.11	⚪ +2.1%	⚪ +2.1%
Q6	2.42	2.42	2.36	2.36	⚪ -2.3%	⚪ -2.3%
Q7	11.95	11.95	11.94	11.94	⚪ -0.1%	⚪ -0.1%
Q8	24.80	24.80	24.34	24.34	⚪ -1.9%	⚪ -1.9%
Q9	38.72	38.72	37.95	37.95	⚪ -2.0%	⚪ -2.0%
Q10	10.32	10.32	10.14	10.14	⚪ -1.7%	⚪ -1.7%
Q11	4.44	4.44	4.39	4.39	⚪ -1.0%	⚪ -1.0%
Q12	6.86	6.86	7.10	7.10	⚪ +3.6%	⚪ +3.6%
Q13	6.80	6.80	6.79	6.79	⚪ -0.2%	⚪ -0.2%
Q14	3.54	3.54	3.56	3.56	⚪ +0.5%	⚪ +0.5%
Q15	7.11	7.11	6.90	6.90	⚪ -2.9%	⚪ -2.9%
Q16	4.77	4.77	4.50	4.50	🟢 -5.7%	🟢 -5.7%
Q17	32.14	32.14	32.12	32.12	⚪ -0.1%	⚪ -0.1%
Q18	33.65	33.65	33.24	33.24	⚪ -1.2%	⚪ -1.2%
Q19	6.68	6.68	6.67	6.67	⚪ -0.1%	⚪ -0.1%
Q20	6.50	6.50	6.59	6.59	⚪ +1.4%	⚪ +1.4%
Q21	46.39	46.39	46.42	46.42	⚪ +0.1%	⚪ +0.1%
Q22	4.99	4.99	4.85	4.85	⚪ -2.8%	⚪ -2.8%
Total	308.08	308.08	306.57	306.57	⚪ -0.5%	⚪ -0.5%

Spark Configuration

Setting	Value
Spark Master	local[*]
Driver Memory	32G
Driver Cores	8
Executor Memory	32G
Executor Cores	8
Off-Heap Enabled	true
Off-Heap Size	24g
Shuffle Manager	CometShuffleManager
Comet Replace SMJ	true

---

Automated benchmark run by dfbench

[mbutrovich]

Thanks @andygrove!