Workaround GCC AVX512 movemask truncation issue

[stefanatwork]

No description provided.

[svenwoop]

Thinking more about this, I think we need a more general fix. The issue is likely visible as mask in GPR has higher bits corrupted, and the movemask is then just a NOP, which reveals the broken higher bits. Could you please try if you can fix the AVX512 movemask implementation in vbollf16_avx512.h by adding the & 0xFFFF there?

[kraszkow]

Thinking more about this, I think we need a more general fix. The issue is likely visible as mask in GPR has higher bits corrupted, and the movemask is then just a NOP, which reveals the broken higher bits. Could you please try if you can fix the AVX512 movemask implementation in vbollf16_avx512.h by adding the & 0xFFFF there?

I had the same idea yesterday, but unfortunately it didn’t work. I also changed the return type from size_t to unsigned short (in addition to adding the mask), but the result was still the same. This fix is really a strange workaround — it only works when implemented exactly the way Stefan did.

[svenwoop]

Ok, then lets merge this workaround. At least we tried a more general fix.

[kraszkow]

Ok, then lets merge this workaround. At least we tried a more general fix.

I’d like to implement and merge the CI job that covers this test case. Then we’ll be able to clearly see that the fix actually works. ETA: today.

Edit: Done (see additional commit to this PR)

[kraszkow]

LGTM