BYOM on STM32N6570-DK: How to handle custom preprocessing/feature extraction with Model Zoo services?

[Sahand]

Hi all — I’ve successfully trained → quantized → deployed one of the audio models from the STM32 AI Model Zoo to an STM32N6570-DK (N6) and can see correct output over the serial port. Now I want to deploy my own model. The model was trained on custom features. I can export to ONNX with fixed input size and compatible ops and I know how to run the quantization + deployment flows from the Model Zoo service.

 

My questions:

1. Preprocessing location

   • The N6 audio example seems to compute only log-mel on-device. Can the Model Zoo service itself be configured to produce my custom features on the target, or is the intended approach to modify the STM32 application C code to implement my feature pipeline before inference?

2. Using ONNX with custom features

   • Anything specific I should set in the user config for ONNX quantization with a non-mel feature front-end (e.g., any flags to disable built-in mel generation assumptions)?

[Julian E.]

Hello @Sahand,

 

1 - I don't know exactly what you mean by custom features, but there are several cases:

• If you want log-melspectrograms but with different parameters (n_fft, hop_length, etc.), you can change them either in user_config.yaml or directly in ai_model_config.h
• If you want a magnitude spectrogram, he can modify, in ai_model_config.h:
  #define CTRL_X_CUBE_AI_MODE_OUTPUT_1 (CTRL_AI_SPECTROGRAM)
  #define CTRL_X_CUBE_AI_PREPROC (CTRL_AI_STFT)
  (Basically, it reuses the speech enhancement preproc). It's possible that some additional changes are needed, I'm not sure. If he wants other spectral features (MFCC, complex spectrogram), you can reuse the functions from the C audio preproc library (in Middlewares/ST/STM32_AI_AudioPreprocessing_Library).
• If you want something else, you will have to program it himself in the C application. We can provide more details if you explain what exactly you want.

2 - ONNX quantization uses a calibration dataset, which is composed of log-melspectrograms in the model zoo, so if you want other features, you will have to perform the quantization himself and provide your custom features in the calibration dataset.

 

Have a good day,

Julian

[Sahand]

Hi Julian,

thanks for your reply, I want to have other sets of features rather than
log-melspectograms. More specifically, I have trained a model using PCEN,
spectral centroid and rolloff, and harmonic mel-spectrogram. Any help would
be appreciated.

Thanks!
Sahand

[Julian E.]

HI @Sahand,

 

Unfortunately, none of the listed processing you listed is available on our C application, you will have to code them yourself.

 

Have a good day,

Julian