v3.3.0 crashes with Rust error -- indefinite length map?

[makew0rld]

In this dir of dasl-testing, the test suite runs uv run main.py libipld to run the Python tests on python-libipld specifically. This works with v3.2.0, but after upgrading it fails:

❯ RUST_BACKTRACE=full uv run main.py libipld

thread '<unnamed>' panicked at src/lib.rs:223:43:
contains length
stack backtrace:
   0:     0x7fe2dcdca9a2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hdcfcb6d4c8489523
   1:     0x7fe2dcdadd93 - core::fmt::write::h8a494366950f23bb
   2:     0x7fe2dcdca37b - std::io::Write::write_fmt::h6556609fca33d0b1
   3:     0x7fe2dcdca803 - std::sys::backtrace::BacktraceLock::print::hb2a626a81e06b2dc
   4:     0x7fe2dcdc9e40 - std::panicking::rust_panic_with_hook::h33ac55f64bbd807d
   5:     0x7fe2dcdf8188 - std::panicking::begin_panic_handler::{{closure}}::h30e7cb89678a57fe
   6:     0x7fe2dcdf80e9 - std::sys::backtrace::__rust_end_short_backtrace::hed60f27456c16ced
   7:     0x7fe2dcdf876c - __rustc[de2ca18b4c54d5b8]::rust_begin_unwind
   8:     0x7fe2dcdacd3f - core::panicking::panic_fmt::h62f63d096dd276af
   9:     0x7fe2dcdaf28a - core::option::expect_failed::h091923fb77e757a1
  10:     0x7fe2dcda39d7 - libipld::decode_dag_cbor_to_pyobject::h80c79c22d9de8a70
  11:     0x7fe2dcda795a - libipld::<impl libipld::decode_dag_cbor::MakeDef>::_PYO3_DEF::trampoline::h1db913fd485e9a26
  12:     0x7fe2dd37631e - _PyEval_EvalFrameDefault
  13:     0x7fe2dd44e2c9 - PyEval_EvalCode
  14:     0x7fe2dd48c88c - <unknown>
  15:     0x7fe2dd48985d - <unknown>
  16:     0x7fe2dd486f58 - <unknown>
  17:     0x7fe2dd486212 - <unknown>
  18:     0x7fe2dd485b83 - <unknown>
  19:     0x7fe2dd483e51 - Py_RunMain
  20:     0x7fe2dd43bbeb - Py_BytesMain
  21:     0x7fe2dce27635 - <unknown>
  22:     0x7fe2dce276e9 - __libc_start_main
  23:     0x55f311930045 - _start
  24:                0x0 - <unknown>

The object that causes the failure is bfff, a valid but indefinite length map.

The offending commit is f988512, with the issue being on line 223 of lib.rs. Instead of decode_len propagating an error, expect is used.

python-libipld/src/lib.rs

Lines 222 to 223 in ff16379

	major::MAP => {
	let len = types::Map::len(r)?.expect("contains length");

Presumably the same issue occurs with arrays.

I don't know enough about Rust to nicely fix this, sorry. Maybe just bringing back decode_len would work?

I would definitely recommend removing expect usage wherever possible, since these kinds of crashes can't be caught from Python code.

Thanks!

[MarshalX]

@vmx would you mind to take a look?

[MarshalX]

I do feel that we should use something like

let len = types::Map::len(r)?
        .ok_or_else(|| anyhow!("contains length"))?;

instead

[MarshalX]

@makew0rld, thank you for your bug report. I am so sorry for this regression. We changed the underlying lib, and the diff was huge. All possible panics should be resolved in the linked PR. Would you mind trying the new build with your test suite?

All the wheel will be attached to this workflow once finished https://github.com/MarshalX/python-libipld/actions/runs/19872167796

[MarshalX]

FYI I've integrated DASL testing as CI step in #97

Upd. v3.3.1 has been released

[makew0rld]

Thanks for the quick fixes! v3.3.1 doesn't cause any panics.

I've spotted another issue however: dasl-testing regressions. Previously, python-libipld passed all 66 dag-cbor tests. Now it fails 26 of them. You can see this on the website: https://hyphacoop.github.io/dasl-testing/

This again comes from the big PR made for v3.3.0. The older v3.2.0 displays a perfect score.

Let me know if there's anything I can help with advice wise or with running dasl-testing.

[MarshalX]

Thank you!

I do see that DASL testing uses deprecated Rust crate as well. It could be worth to replace it with cbor4ii one as we did it here.

Deprecated libipld crate recommends to migrate to ipld-core. But inside there is serde_ipld_dagcbor crate which uses cbor4ii at the end. So technically DASL testing could pick any of these crates

I do think many issues came from upstream.

Upd. Oh. I do see serde_ipld_dagcbor in DASL! And the errors almost aligns 1:1. Only "valid CID with short tag" fails additionally on Python side

[makew0rld]

DASL testing is totally isolated for at least each programming language, often for each project as well. So the fact it uses a Rust library in one place should be irrelevant.

My broader point is that v3.2.0 of this lib passed all the dag-cbor tests, and the latest v3.3.1 fails many of them. So there has been some regressions.

Most of these failures are common and related to minimal encoding. This is not a huge issue, although it would be great to see it fixed!

There is one more important regression though, which is CIDs seemingly can't be roundtripped anymore:

Proof:

❯ uv run --with libipld==3.2.0 python
Installed 1 package in 4ms
Python 3.13.7 (main, Aug 15 2025, 12:34:02) [GCC 15.2.1 20250813] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libipld
>>> libipld.encode_dag_cbor(libipld.decode_dag_cbor(bytes.fromhex("d82a582500015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03"))).hex()
'd82a582500015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03'

❯ uv run --with libipld==3.3.1 python
Installed 1 package in 5ms
Python 3.13.7 (main, Aug 15 2025, 12:34:02) [GCC 15.2.1 20250813] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libipld
>>> libipld.encode_dag_cbor(libipld.decode_dag_cbor(bytes.fromhex("d82a582500015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03"))).hex()
'582500015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03'

Anyway, I'll let you handle things. I appreciate all the quick responses. Let me know if I can help with anything!

[MarshalX]

@makew0rld, let me clarify things. This Python library (named python-libipld in DASL) is a wrapper around the Rust library. This package does not decode and encode CBOR directly on the byte level. The only thing that this package implements on its own is CAR decoding. Which is not covered by DASL. This library uses the same Rust crate as serde_ipld_dagcbor under the hood. In v3.3.0, we changed the base Rust library from libipld (also covered by DASL) to Rust cbor4ii.

TL;DR: serde_ipld_dagcbor and python-libipld use the same cbor4ii Rust crate under the hood

As mentioned, yes, there is a valid CID with short tag test that fails additionally, and I will take a look 100% :)

[makew0rld]

Thanks, I understand. Sorry I misunderstood you.

libipld, despite being deprecated, passed a lot of the tests. So it makes sense python-libipld would pass fewer tests in switching from libipld to cbor4ii.

Best of luck with the bug fixing!