SharpToken.GptEncoding.GetEncoding(o200k_base) takes minutes

[vovanb]

Hi
I am using your library in MAUI android. For some reason method
SharpToken.GptEncoding.GetEncoding(o200k_base)
first time takes minutes to respond
What can be issue?

[dmitry-brazhenko]

Hey!

Sounds interesting. Could you share some more details please

1. Text length
2. android ver
3. device

Probably could you share apk to check it?
My guess is that it takes significant time just to read tokenizer dict, but not sure about it.