fix: spark version in integration_tests

[feniljain]

Issue

• Spark 3.5.3 seems to have disappeared from here, there's 3.5.4 instead, so upgrading to that

Question

• Is there any way to keep auto-updating this? Would we have to do manual version bumps everytime?

[kevinjqliu]

LGTM, CI passes

For context, https://dlcdn.apache.org/spark hosts the latest version and removes the older versions. In PyIceberg we had to switch back to use archive.apache.org which hosts all versions

apache/iceberg-python#1523

[feniljain]

Ohh, we recently moved away from archive here

Tho comment posted by you seems more recent and by original author (@Fokko) of above PR itself, should we change it back to archive in that case?

[kevinjqliu]

PyIceberg is currently blocked on using 3.5.4 so we have to use archive.apache.org for the older version.

If we use dlcdn.apache.org, we would run into this issue on every new release. But it seems like the reason we moved away from archive.apache.org is because its unstable.

iceberg-rust is able to use 3.5.4 so I'm happy to merge this change.

[liurenjie1024]

Thanks @feniljain for this fix.

[liurenjie1024]

PyIceberg is currently blocked on using 3.5.4 so we have to use archive.apache.org for the older version.

If we use dlcdn.apache.org, we would run into this issue on every new release. But it seems like the reason we moved away from archive.apache.org is because its unstable.

iceberg-rust is able to use 3.5.4 so I'm happy to merge this change.

Or should we consider using spark's container image directly?

[kevinjqliu]

@liurenjie1024 yea i was thinking of that too, i opened apache/iceberg-python#1527 on the python side to track this