Skip to content

docs: Update confs to bypass Iceberg Spark issues #2166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
parthchandra merged 4 commits into from
Aug 19, 2025
Merged

docs: Update confs to bypass Iceberg Spark issues #2166

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ea198d9
Update confs to bypass Iceberg Spark issues
hsiang-c Aug 16, 2025
4cf0b14
Update iceberg.md
hsiang-c Aug 18, 2025
47f85a3
Users can diable Spark's AQE as well
hsiang-c Aug 18, 2025
63d9b71
Let users turn off AQE or Comet's broadcastExchange
hsiang-c Aug 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions docs/source/user-guide/iceberg.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,11 +80,13 @@ $SPARK_HOME/bin/spark-shell \
--conf spark.sql.catalog.spark_catalog.type=hadoop \
--conf spark.sql.catalog.spark_catalog.warehouse=/tmp/warehouse \
--conf spark.plugins=org.apache.spark.CometPlugin \
--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
--conf spark.comet.exec.shuffle.enabled=false \
--conf spark.sql.iceberg.parquet.reader-type=COMET \
--conf spark.comet.explainFallback.enabled=true \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=2g
--conf spark.memory.offHeap.size=2g \
--conf spark.comet.use.lazyMaterialization=false \
--conf spark.comet.schemaEvolution.enabled=true
```

Create an Iceberg table. Note that Comet will not accelerate this part.
Expand Down Expand Up @@ -138,4 +140,11 @@ scala> spark.sql(s"SELECT * from t1").explain()
== Physical Plan ==
*(1) CometColumnarToRow
+- CometBatchScan spark_catalog.default.t1[c0#26, c1#27] spark_catalog.default.t1 (branch=null) [filters=, groupedBy=] RuntimeFilters: []
```
```

## Known issues
- We temporarily disable Comet when there are delete files in Iceberg scan, see Iceberg [1.8.1 diff](../../../dev/diffs/iceberg/1.8.1.diff) and this [PR](https://github.com/apache/iceberg/pull/13793)
- Iceberg scan w/ delete files lead to [runtime exceptions](https://github.com/apache/datafusion-comet/issues/2117) and [incorrect results](https://github.com/apache/datafusion-comet/issues/2118)
- Enabling `CometShuffleManager` leads to [runtime exceptions](https://github.com/apache/datafusion-comet/issues/2086)
- Spark Runtime Filtering isn't [working](https://github.com/apache/datafusion-comet/issues/2116)
- You can bypass the issue by either setting `spark.sql.adaptive.enabled=false` or `spark.comet.exec.broadcastExchange.enabled=false`