Unable to update Unicode version to `16.0.0`

[adithyaov]

I'm trying to upgrade Unicode to 16.0.0
On running ./ucd generate (after ./ucd/download) I'm greeted with this:

Too many script extensions: 271

Interesting code blocks to look at:

UCD2Haskell/Modules/ScriptExtensions.hs:150

        encodedExtensions :: Map.Map (NE.NonEmpty BS.ShortByteString) Word8
        encodedExtensions = let len = length extensionsList in if len > 0xff
            then error ("Too many script extensions: " <> show len)
            else Map.fromList (zip extensionsList [0..])

        -- Encode single script as their script value
        extensionsList = singleScriptExtensions
                      <> Set.toList multiScriptExtensions

It looks like
length (singleScriptExtensions <> Set.toList multiScriptExtensions)
is > 271.

UCD2Haskell/Modules/ScriptExtensions.hs:141

        singleScriptExtensions = pure . getScriptAbbr <$> scripts
        singleScriptExtensionsSet = Set.fromList singleScriptExtensions
        multiScriptExtensions :: Set.Set (NE.NonEmpty BS.ShortByteString)
        multiScriptExtensions = Set.fromList (Map.elems extensions)
                                Set.\\ singleScriptExtensionsSet

singleScriptExtensionsSet cannot be more than 204 as:

$ cat data/16.0.0/ucd/ScriptExtensions.txt | grep -v '^\s*#' | grep -v '^\s*$' | wc -l
204

So multiScriptExtensions bumps the value up?

I've not investigated this further. We use genEnumBitmapShamochu to generate
the bitmap, and Shamochu restricts the number of elements to be < 0xff. So by
design, we don't allow extensionsList to have more than 0xff elements.

This is a bug if there SHOULD NOT be more than 0xff extensions, or this is a
LIMITATION of the generator if we don't support more than 0xff extensions.

I'm not sure how to fix this as I don't fully understand the problem.

@wismill could you please suggest how I should proceed?

[wismill]

I have not touched this for a long time 😅.

This is a bug if there SHOULD NOT be more than 0xff extensions, or this is a
LIMITATION of the generator if we don't support more than 0xff extensions.

This is the current limitation: I tried to compress the data as much as possible, and it was fine using 8 bits with the previous releases of Unicode. However, the new scripts in 16.0 are incompatible with the previous optimization. So the obvious fix is to encode the script extensions on 16 bits rather than 8. This does not seems completely trivial though, because IIRC, most of the code relies on enums being encodable on 8 bits. We would need to write alternative versions of the relevant functions to make it work on 16 bits.

[adithyaov]

Hi @wismill

Could you please let me know if I'm going about this in the right direction:

The shamochu library provides lossless compression.

Looking at:

compress
    :: forall e. (FiniteBits e, Unbox e, Ord e)
    => NonEmpty Word
    -> [Word]
    -> Vector e
    -> CompressedBlob e

It does not look like this has a size-based restriction on what it can
compress. In our case, we pass in a Vector Word8, but we can just as well pass a
Vector Word16.

That said, there is already a more flexible bitmap generator:

genBitmapShamochu ::
     HasCallStack
  => String
  -> NE.NonEmpty Word
  -- ^ Chunk size stage 1
  -> [Word]
  -- ^ Chunk size stage 2
  -> [Int]
  -> ShamochuCode

as opposed to,

genEnumBitmapShamochu
  :: forall a. (HasCallStack, Bounded a, Enum a, Eq a, Show a)
  => String
  -- ^ Function name
  -> Bool
  -- ^ If true, use raw Int
  -> NE.NonEmpty Word
  -- ^ Chunk size stage 1
  -> [Word]
  -- ^ Chunk size stage 2
  -> (a -> Word8)
  -- ^ Conversion
  -> (a, BB.Builder)
  -- ^ Value for planes 15-16
  -> (a, BB.Builder)
  -- ^ Default value
  -> [a]
  -- ^ List of values to encode for planes 0 to 3
  -> [a]
  -- ^ List of values to encode for plane 14
  -> ShamochuCode

Would the implementation still be correct if I use genBitmapShamochu over
genEnumBitmapShamochu?

In this case, the Enums would be encoded in 64 bits. There probably will be some
cascading changes, which I'll also have to fix.

[wismill]

The shamochu library provides lossless compression.

Looking at:

compress
:: forall e. (FiniteBits e, Unbox e, Ord e)
=> NonEmpty Word
-> [Word]
-> Vector e
-> CompressedBlob e

It does not look like this has a size-based restriction on what it can compress. In our case, we pass in a Vector Word8, but we can just as well pass a Vector Word16.

That’s correct.

That said, there is already a more flexible bitmap generator:

genBitmapShamochu ::
     HasCallStack
  => String
  -> NE.NonEmpty Word
  -- ^ Chunk size stage 1
  -> [Word]
  -- ^ Chunk size stage 2
  -> [Int]
  -> ShamochuCode

[…]

Would the implementation still be correct if I use genBitmapShamochu over genEnumBitmapShamochu?

The issue is that genBitmapShamochu actually encodes Bool, not other Ints. The last parameter is the list of indexes of the True values.

In this case, the Enums would be encoded in 64 bits.

While using more space seems inevitable and should not block updating to Unicode 16.0, a 64-bit (×8) encoding seems excessive here, so we should limit to 16 bits (×2)

Do you have an agenda for the update? I would like to take a look at it, but I confess that my Haskell got rusty.

It seems that the first step would be to better document all those functions 😅.

[adithyaov]

While using more space seems inevitable and should not block updating to Unicode 16.0, a 64-bit (×8) encoding seems excessive here, so we should limit to 16 bits (×2)

I agree. I was trying to confirm the usage of genBitmapShamochu.

Do you have an agenda for the update?

Please see: #137

I would like to take a look at it, but I confess that my Haskell got rusty.
It seems that the first step would be to better document all those functions 😅.

Please let me know if you need any help. Updating things as pseudo-code will
also help a lot. I can take care of getting it to compile.

[wismill]

Do you have an agenda for the update?

Please see: #137

I meant do you have a dead line?

[adithyaov]

I meant do you have a dead line?

Not really. There's no urgency per se.
But the sooner, the better, I suppose. :-)

[wismill]

I finally found time to work on this, see #139. While I am confident that the shamochu package works correctly, I am still a bit uneasy there could be still some hard-coded bits in ucd2haskell that assume all values can be encoded as Word8. Tests are exhaustive and pass against ICU though, so I am maybe a bit paranoid 😅