AI Core inconsistent inference results

[MichalPesko]

Hello, I am developing a Dog Activity Recognition app with CubeAI on STM32WB55 usign AI Core 9.0.0.

So far, I have been following https://wiki.st.com/stm32mcu/wiki/AI:How_to_perform_motion_sensing_on_STM32L4_IoTnode . I am using the same sensor and ODR, same AI input size, same Model structure, but my own dataset from Google Colab.

Model Structure:

Accuracy in Google Colab: 94 %.

After getting almost always the same "wrong" result in STMCubeIDE, I did the following:

• I performed validation on target in STMCubeIDE (the accuracy is similar to Google Colab, which is good:(acc=95.13%, rmse=0.145657539, mae=0.044287317, l2r=0.261498094, nse=0.905, cos=0.968
• I tested a couple of inputs manually to compare with result in Colab and this is where the problem seems to be (Test1 is OK), the results do not match quite a lot:

How this can be? The model in Colab is the same as STMCubeIDE.

 

Side question: does AI Core transform Conv1D layers into Conv2D, as seen in the first picure? Can this possibly be the reason?

 

Thank you very much!

[fauvarque.daniel]

You can't expect the results to be binary the same as on one side you have code in python and on the target you have another implementation of the kernels in C.

If the accuracy for the validation on the target is ok with a significant dataset then the differences you are seeing are probably in the 5% of accuracy loss.

That said a cos of 0.968 on a significant dataset is not so good, we are expecting more in the range of 0.99 to 1.

Maybe you can share your model for a closer look at any potential issue.

Regards

[unsigned_char_array]

---

@fauvarque.daniel wrote:

You can't expect the results to be binary the same as on one side you have code in python and on the target you have another implementation of the kernels in C.

---

I assume this is due to model quantization or other form of transforming so the model can be deployed to the MCU, is this correct? Is there a way to simulate using the same kernel as the target? Is there a way to retrain using the same kernel to improve accuracy? To me such a huge difference is unacceptable and I wonder if there is a way to make them closer.

[fauvarque.daniel]

You can verify the accuracy by running the model on the host, this will use the same C kernels and compare the resuts with a run in python.

The important point is to have a relevant dataset when running the validation. You can run the validation with random data but with quantized networks the results may not be the real ones.