S3: The checksum of the sharding index is invalid.

[dominikl]

A zarr which works perfectly fine using the FileStore does not work when uploaded to S3. Here's an example:
https://uk1s3.embassy.ebi.ac.uk/idr/share/ome2024-ngff-challenge/idr0066/ExpB_whole_cns_MIP.ome.zarr/0

Download it with ome_zarr (pip install ome-zarr), aws won't work because no List permission:

ome_zarr download https://uk1s3.embassy.ebi.ac.uk/idr/share/ome2024-ngff-challenge/idr0066/ExpB_whole_cns_MIP.ome.zarr/0 --output ExpB_whole_cns_MIP.ome.zarr

Example code: https://gist.github.com/dominikl/3719fd5c3a44a159a68aec9d64724292

This will result in (I've just added some logging):

Testing local...
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
store: file:///Users/dom/testbf
group: <v3.Group {file:///Users/dom/testbf/ExpB_whole_cns_MIP.ome.zarr/0}>
First res: <v3.Array {file:///Users/dom/testbf/ExpB_whole_cns_MIP.ome.zarr/0/0} (1, 2, 1, 2738, 9357) UINT16>
Read array: 134 139 141 142 139 135 131 ...
Testing S3...
store: s3://idr/null
Handle: s3://idr/null/share/ome2024-ngff-challenge/idr0066/ExpB_whole_cns_MIP.ome.zarr/0
group: <v3.Group {s3://idr/null/share/ome2024-ngff-challenge/idr0066/ExpB_whole_cns_MIP.ome.zarr/0}>
First res: <v3.Array {s3://idr/null/share/ome2024-ngff-challenge/idr0066/ExpB_whole_cns_MIP.ome.zarr/0/0} (1, 2, 1, 2738, 9357) UINT16>
java.lang.RuntimeException: dev.zarr.zarrjava.ZarrException: The checksum of the sharding index is invalid. Stored: 524201981 Computed: 424531239
        at dev.zarr.zarrjava.core.Array.lambda$1(Array.java:325)
        at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
        at dev.zarr.zarrjava.core.Array.read(Array.java:292)
        at dev.zarr.zarrjava.core.Array.read(Array.java:233)
        at com.example.zarrdemo.TestV3.loadArray(TestV3.java:81)
        at com.example.zarrdemo.TestV3.testS3(TestV3.java:56)
        at com.example.zarrdemo.TestV3.main(TestV3.java:37)
Caused by: dev.zarr.zarrjava.ZarrException: The checksum of the sharding index is invalid. Stored: 524201981 Computed: 424531239
        at dev.zarr.zarrjava.v3.codec.core.Crc32cCodec.decode(Crc32cCodec.java:40)
        at dev.zarr.zarrjava.core.codec.CodecPipeline.decode(CodecPipeline.java:113)
        at dev.zarr.zarrjava.v3.codec.core.ShardingIndexedCodec.decodeInternal(ShardingIndexedCodec.java:203)
        at dev.zarr.zarrjava.v3.codec.core.ShardingIndexedCodec.decodePartial(ShardingIndexedCodec.java:253)
        at dev.zarr.zarrjava.core.codec.CodecPipeline.decodePartial(CodecPipeline.java:94)
        at dev.zarr.zarrjava.core.Array.lambda$1(Array.java:313)

I'm pretty stuck there unfortunately.

[brokkoli71]

Hi, thanks for the detailed bug report.
I could reproduce this on main but on the branch of #37 the code runs without problems.
I think this is related to a range mismatch in S3Store.get fixed in #37. We are planning to merge this today and then create a release