Arcilator crashes in Upload Release Artifacts CI

[fabianschuiki]

It looks like there are two configs of the Upload Release Artifacts CI job where check-circt fails on three arcilator tests:

https://github.com/llvm/circt/actions/runs/19579832326/job/56099287786

Not sure what changed since the last run of this CI action that causes the crash. @jpienaar could this be related to the additional arguments that are now getting passed to the simulation if run in JIT mode?

[jpienaar]

Could very well be yes. emit-values wasn't changed but could be some overlap in code. Surprised it fails in this manner (on some but not all). I'll try to take a look ASAP.

[fzi-hielscher]

The failing build is linking against musl libc, isn't it?

Considering which tests are failing, I'm more inclined to believe d7ec099 is the problem. Looks like portable libc calls are even harder than I already thought. Musl hasn't been part of my tests so far...

[jpienaar]

Re musl libc part, should we defer until that can be checked? And/or easiest way to repro with musl (if related to that PR and musl, I think just repeated printf calls could be used to simulate the same result while using calls as before)

[seldridge]

Yeah, that was musl on Alpine in the "actually static binary" build flow.

I was inclined to believe that this may not be easily fixable and went ahead with moving these builds to Rocky Linux 8 and being "mostly static". I already merged this change onto main here: 7982bdf You'd need to revert this in order to test with the workflow dispatch trigger.

If you have line of sight to fixing this, that would be great! Otherwise, I'm beginning to think that the musl builds were a good idea, but likely unnecessary. These primarily exist to provide a firtool for Chisel users that are on old EDA machines with old GLIBC versions. This usually means RHEL. The musl builds have been a huge headache due to weird choices in default thread stack sizes, a poor implementation of malloc requiring switching to mimalloc, and now whatever this is.

The underlying requirement is that we provide a firtool that is guaranteed to work on most EDA machines. If we only support RHEL 8 (which is what it will be after my change), I think that's defensible. RHEL 7 is in Extended Life Cycle support and CentOS 7 is EOL at this point.

[fzi-hielscher]

I don't have a good explanation yet other than musl disliking the new format specifier for some reason:
@_arc_sim_emit_arg1 = internal constant [15 x i8] c"arg1 = %08llx\0A\00"
vs.
@_arc_sim_emit_full_arg1 = internal constant [12 x i8] c"arg1 = %zx\0A\00"

I'll try to repro the failure in a local container, but it is going to take a while.

[seldridge]

@fzi-hielscher: Nice work. It looks like this is the very first list of GLIBC differences: https://wiki.musl-libc.org/functional-differences-from-glibc

I think you could probably try to just change that and land it.

[fabianschuiki]

Yikes, that's very subtle! 😁

[fzi-hielscher]

Yikes, that's very subtle!

I'm afraid it is too subtle for me at the moment. 😵

glibc supports some alternate incorrect format specifiers, like %Ld as an alias for %lld. musl does not support these, and returns an error upon encountering them.

The way I'm parsing this %08llx should work*, while %08Lx would be glibc only.

*Unless we are pre C99

[fzi-hielscher]

Ah, sorry for the confusion. When I wrote "other than musl disliking the new format specifier for some reason" that was a hypothesis, not an observation. I've now tested it (without using arcilator) and it works just fine, as you'd expect.

I've also now realized that all three failing test have been added between this release and the previous one. I strongly suspect that arc.sim.emit has never been working in the musl build and we just didn't notice. It is probably simply due to printf not being linked into the arcilator binary since it is not referenced outside of the IR. But again, just a hypothesis. 😅 I'll try to confirm it tomorrow.

[seldridge]

Thank you for digging into it! I'm still fine to pull the plug on the muslc builds and switch to Rocky Linux 8. This would likely be better for all the non-static builds, too, as then these would be directly usable in most EDA tools builds.

[fzi-hielscher]

I've now got a static build that should be close enough to the one in the CI. Bad news: The MLIR Execution Engine won't initialize. Internally it is calling dlopen(NULL); to get a handle on the symbols of the current process. This inevitably fails in musl's stub implementation.

I managed to bodge a config flag making it start anyway, see the attached patch file. My knowledge of the JIT engine is practically non existent, but I hope that this leaves us with a functional runner that is simply unaware of any external symbols. Naturally, we now have to manually provide a pointer to all functions we intend to call from the IR (see the other attached patch), but I had considered doing that anyway to avoid any visibility related problems. With the two patches applied, it seems to be working. However, I don't think the change would be too well received in upstream. If we are determined to provide a musl compatible arcilator binary with JIT support, we could consider writing our own wrapper around LLJIT.

ee_noDlopen_hack.patch

arcilator.patch

[fabianschuiki]

If musl doesn't support dynamic object loading, do we ever have a chance to get a working musl build of Arcilator?