The int8 input output is unknown type at ml toolkit by FVP

[kris-himax]

Hi,
I generated the mobilenet V2 .pte file input output is int8 by the python script.

python3 -m examples.arm.aot_arm_compiler --model_name=mv2 --target=ethos-u55-64 --delegate --quantize --config=himax_vela.ini --system_config=My_Sys_Cfg --memory_mode=My_Mem_Mode_Parent --output=mv2_quant_himax_ini_opt_size_vela_4_5_0_inout_in8.pte

Build by ml toolkit, but the inputTensor->Type() are all unknown.

cc @freddan80 @per @zingo @oscarandersson8218 @digantdesai

[zingo]

Thanks we will have a look

[gggekov]

@kris-himax i think the problem is that in the AoT part, you generate a pte file with the QuantizeInputs / QuantizeOutputs passes. In other words, the pte file expects int8 numbers as input/output. I suspect that the pre-processing in MLEK is for a variant of MV2 with FP32 input and Q/DQ nodes around the Ethos-U custom delegate. Can you please double check that when you use QuantizeInputs / QuantizeOutputs passes to generate a pte, in the C++ application logic you populate int8 data(and not FP32)?
Thanks,
George

[kris-himax]

@gggekov
I did not change any code at the MLEK but only change the model at the usecase.cmake.

   if (ETHOS_U_NPU_ENABLED)
        string(TOLOWER ${ETHOSU_TARGET_NPU_CONFIG} _NPU_CFG_ID)
        set(DEFAULT_MODEL_PATH          ${DEFAULT_MODEL_DIR}/mv2_quant_himax_ini_opt_size_vela_4_5_0_inout_in8.pte)
        set(DEFAULT_ACTIVATION_BUF_SZ   0x00200000)

I add the printf at PrepareInputTensors() to see the tensorMeta.get().scalar_type(),and it show it is 1 which I think is Char type.

printf("@@@@@@@@@@@@@tensorMeta.get().scalar_type(): %d\r\n",tensorMeta.get().scalar_type());

So I make sure when I use QuantizeInputs / QuantizeOutputs passes to generate a pte, in the C++ application logic I populate int8 data.

But I can load the same model and see the correct type at my local MCU.

Thanks,
Kris

[gggekov]

Thanks @kris-himax .
Is this an issue only when you apply the QuantizeInputs / QuantizeOutputs passes? Would the application work if you generate a pte with FP32 input + Quant to int8 node, Ethos custom delegate and dequant to FP32(that is the default ExecuTorch behaviour) ?

Also, we recently hosted a simple img classification application showing the bare minimum steps you need for an e2e application. Perhaps that's something that may be useful.

The Corstone-320 application is in https://github.com/pytorch/executorch/tree/main/examples/arm/image_classification_example/runtime . You can try to plug your pte, but you would need to populate the input tensor with your values. The input tensor is populated here - https://github.com/pytorch/executorch/blob/main/examples/arm/image_classification_example/runtime/main.cpp#L176
where the image_data array is produced by the https://github.com/pytorch/executorch/blob/main/examples/arm/image_classification_example/runtime/rgb_to_array.py script, called in https://github.com/pytorch/executorch/blob/main/examples/arm/image_classification_example/runtime/CMakeLists.txt#L142
For our application, we use Deit_Tiny and it targets the Corstone-320. If you have a smaller model that fits on the Corstone-300 or on your SoC, you can easily run the pte on a U55 as well. The runtime logic is exactly the same irrespective of whether you have U55 or U85.