the result of onnx model validate on target has two different output, why?

[cxf]

hi

I have validate on the STM32N6 board by onnx model in STM32MX,but i have two different output ,like this:

 m_outputs_1: (10, 3)/float64, min/max=[-282.578064, 481.370483], mean/std=[-2.619613, 298.301398], output  m_outputs_2: (10, 1)/float64, min/max=[0.000000, 0.000000], mean/std=[0.000000, 0.000000], node_139  c_outputs_1: (10, 1, 1, 3)/float32, min/max=[-4.547175, 5.813805], mean/std=[0.008044, 4.321293], output  c_outputs_2: (10, 1, 1, 1)/float32, min/max=[0.000800, 0.000800], mean/std=[0.000800, 0.000000], node_139

the  m_outputs_1 is error, and c_outputs_1 is correct,why? what can lead to this problem?

please help me ,thanks very much!!!

I have upload the report.txt and onnx model in the attach.

[Julian E.]

Hello @cxf,

 

Can you give me more context about what you have done exactly:

• What version of X Cube AI do you use?
• Which board: Nucleo or DK n6?
• Did you use custom data for validation or the random one by default when you do a validate on target?
• Are you using STM32CubeIDE as the IDE when doing the Validation on target or do you use another IDE?

 

I did test with the Nucleo N6 and N6 DK board, with version 10.1.0 and 10.2.0 of X Cube AI.

I am using STM32CubeIDE and windows, but I never replicated your issue.

 

Here is an example of report I get, which is correct:

Saving validation data...
 output directory: C:\ST\STEdgeAI\2.2\scripts\N6_scripts\st_ai_output
 creating C:\ST\STEdgeAI\2.2\scripts\N6_scripts\st_ai_output\network_val_io.npz
 m_outputs_1: (10, 3)/float64, min/max=[-282.578064, 481.370483], mean/std=[-2.619613, 298.301398], output
 m_outputs_2: (10, 1)/float64, min/max=[0.000000, 0.000000], mean/std=[0.000000, 0.000000], node_139
 c_outputs_1: (10, 1, 1, 3)/float32, min/max=[-282.578094, 481.370422], mean/std=[-2.619609, 298.301392], output
 c_outputs_2: (10, 1, 1, 1)/float32, min/max=[0.000000, 0.000000], mean/std=[0.000000, 0.000000], node_139

 
Computing the metrics...

 Cross accuracy report #1 (reference vs C-model)
 ----------------------------------------------------------------------------------------------------
 notes: - data type is different: r/float64 instead p/float32
 - ACC metric is not computed ("--classifier" option can be used to force it)
 - the output of the reference model is used as ground truth/reference value
 - 10 samples (3 items per sample)

 acc=n.a. rmse=0.000060476 mae=0.000043488 l2r=0.000000203 mean=-0.000004 std=0.000061 nse=1.000000 cos=1.000000 

 Cross accuracy report #2 (reference vs C-model)
 ----------------------------------------------------------------------------------------------------
 notes: - data type is different: r/float64 instead p/float32
 - the output of the reference model is used as ground truth/reference value
 - 10 samples (1 items per sample)

 acc=n.a. rmse=0.000000000 mae=0.000000000 l2r=0.000000000 mean=0.000000 std=0.000000 nse=1.000000 cos=1.000000 


Evaluation report (summary)
---------------------------------------------------------------------------------------------------------------------------------------------------
Output acc rmse mae l2r mean std nse cos tensor 
---------------------------------------------------------------------------------------------------------------------------------------------------
X-cross #1 n.a. 0.000060476 0.000043488 0.000000203 -0.000004 0.000061 1.000000 1.000000 'output', 10 x f32(1x3), m_id=[110] 
X-cross #2 n.a. 0.000000000 0.000000000 0.000000000 0.000000 0.000000 1.000000 1.000000 'node_139', 10 x f32(1x1), m_id=[112] 
---------------------------------------------------------------------------------------------------------------------------------------------------

 

Have a good day,

Julian

 

[cxf]

hello @Julian E. 

 

Thank you for your suggestion very much . Now I list my  development environment like this:

1: X Cube AI 10.1.0

2:STM32N6570-DK  board

3:validation input - random numbers 

4:STM32CubeIDE 1.18.1 on windows10

5:STM32cubeMX 6.15.0 

 

As you result, i have two question:

1: your's m_outputs_1 and c_outputs_1 are almost the same, so output have been stabilized‌，

This is a progress.  But they are error results, because the min<-100, max>100, the correct result is min/max @ [-30,30]

 

2: In my STM32MX, when i click check update,the X Cube AI 10.1.0 is the highset version,

where is the 10.2.0?

 

Thanks!

[Julian E.]

Hello @cxf,

 

As of now, you need to install X Cube AI 10.2.0 from local files. You can download it here:

X-CUBE-AI - AI expansion pack for STM32CubeMX - STMicroelectronics

And then install it by clicking "From Local..." in the package install manager in STM32CubeMX.

 

I didn't understand your other point:

your's m_outputs_1 and c_outputs_1 are almost the same, so output have been stabilized‌，

This is a progress.  But they are error results, because the min<-100, max>100, the correct result is min/max @ [-30,30]

Where does the -100,100 comes from and more importantly why do you say that -30,30 is the correct value?

 

Have a good day,

Julian

[hamitiya]

hello @cxf 

it seems that it is not yet available on STM32CubeMX updater. You can install it manually by downloading packages from this link:

X-CUBE-AI - AI expansion pack for STM32CubeMX - STMicroelectronics

 

Best regards,

Yanis

[SlothGrill]

Hello
Can you paste the end of the traceback in your screenshot above ? please?

(i.e. the above exception was the direct cause .... up to the end ?)

+ if this does not help (i.e. if the cubeide path mentionned here is the one your are expecting), you can raise the verbosity of scripts by setting .setLevel(logging.DEBUG) in both n6_utils_pkg/config_reader.py (line 14) oand n6_utils_pkg/cubeIDE_toolbox.py (line 15), this will help find the issue.

 

Possible fix is to remove your  "cubeide_path": "E:/ST/STM32CubeIDE_1.18.1/STM32CubeIDE", line from the config file and see if it works better. (though it would be better to understand what's not good about your config here..)

[cxf]

Hi @SlothGrill 

Thank you for your help very much!

Now, i have follow your advice ,and modify the two py like this:

 

 

and the cubeide_path has been set to "",when I run the n6_loader.py, result can be get in the attach.

please help me thanks !

 

 

 

[SlothGrill]

hello
Thanks for the log could you further help to find what is going on:

• In a shell, if you go to `...\X-CUBE-AI\10.2.0\scripts\N6_scripts` , what is the output of `which stm32cubeide` ?
• Based on the outputs, i guess the cubeide_path has been set to ".", i'm not sure what could have led to this... do you have another line with cubeide_path in your config.json ? 
• Can you try running python n6_loader --help
• You can also try to force the cubeidepath on the commandline and see if the behaviour is different (it should not be)
• did you try to remove the cubeide_path line from your config.json ?

Do you know what shell you are running by default ? 

Do you run a clean install of stedgeai or did you tweak it in some way (or eg. migrate from an old version to a new one ?)

 

Sorry, from the outputs this is not clear what is going on :\

Best regards.

 

[cxf]

Hi @SlothGrill 

 

Thank you for your help very much!!!

 

Now,I modified the config.json in the \X-CUBE-AI\10.2.0\scripts\N6_scripts  like this:

{

"compiler_type": "gcc",

    "gdb_server_path": "E:/ST/STM32CubeIDE_1.18.1/STM32CubeIDE/plugins/com.st.stm32cube.ide.mcu.externaltools.stlink-gdb-server.win32_2.2.100.202501151542/tools/bin/",

    "gcc_binary_path": "E:/ST/STM32CubeIDE_1.18.1/STM32CubeIDE/plugins/com.st.stm32cube.ide.mcu.externaltools.gnu-tools-for-stm32.13.3.rel1.win32_1.0.0.202411081344/tools/bin/",

    "iar_binary_path": "",

    "objcopy_binary_path": "E:/ST/STM32CubeIDE_1.18.1/STM32CubeIDE/plugins/com.st.stm32cube.ide.mcu.externaltools.gnu-tools-for-stm32.13.3.rel1.win32_1.0.0.202411081344/tools/bin/arm-none-eabi-objcopy.exe",

    "cubeProgrammerCLI_binary_path": "E:/Program Files/STMicroelectronics/STM32Cube/STM32CubeProgrammer/bin/STM32_Programmer_CLI.exe",

    "cubeide_path": "E:/ST/STM32CubeIDE_1.18.1/STM32CubeIDE"

}

 

and  get the same two results by  Julian E.'s five steps,like this:

By this way,I think I have validate on target successfully.

But when I go to STM32CubeMx,and click the validate on target,and check the Enabled,I still get the previous issues ，like this:

when I get the previous issues, I found that the config.json has been modified to like this:

  {

  "gdb_server_path": "E:/ST/STM32CubeIDE_1.18.1/STM32CubeIDE/plugins/com.st.stm32cube.ide.mcu.externaltools.stlink-gdb-server.win32_2.2.100.202501151542/tools/bin/",

    "gcc_binary_path": "E:/ST/STM32CubeIDE_1.18.1/STM32CubeIDE/plugins/com.st.stm32cube.ide.mcu.externaltools.gnu-tools-for-stm32.13.3.rel1.win32_1.0.0.202411081344/tools/bin/",

    "objcopy_binary_path": "E:/ST/STM32CubeIDE_1.18.1/STM32CubeIDE/plugins/com.st.stm32cube.ide.mcu.externaltools.gnu-tools-for-stm32.13.3.rel1.win32_1.0.0.202411081344/tools/bin/arm-none-eabi-objcopy.exe",

    "cubeide_path": "",

    "compiler_type": "gcc",

    "cubeProgrammerCLI_binary_path": "E:/Program Files/STMicroelectronics/STM32Cube/STM32CubeProgrammer/bin/STM32_Programmer_CLI.exe",

    "iar_binary_path": ""

}

 As I have validate on target successfully by the first way(above-mentioned),

why it can generate this problem by use STM32CubeMX?

Is there any differences between two way?

 

Thanks!!!

I put the shell information  into the attach.

[SlothGrill]

Hello
I'm not sure to fully understand, but there is definitely a problem with your last config.json: setting "cubeide_path" to "" is not similar to fully removing the line (in the way the config file is parsed).

Setting it to "", indicates to the tool that you expect to find cubeide in your current directory, then when trying to find gdb_server in your current directory, everything fails.

So, if this is x-cube-ai who erased your config.json, then this is problematic :), but the main thing is that you now know how to perform a validation on target after editing your config file.

 

I don't really understand if there are still issues pending:

• After editing the config.json, you manage to do a validation on target through X-Cube-AI
• so, you can now perform validations on target with X-Cube-AI, right ?
• Could you please "summarize" the "two ways" in a synthetic manner ? 

Thanks

Cheers.

 

[cxf]

Hi  @SlothGrill 

Thank you for your help very much!!!

The two methods  are like this:

First method:

1. Go to: C:\Users\<your_user>\STM32Cube\Repository\Packs\STMicroelectronics\X-CUBE-AI\10.2.0\scripts\N6_scripts\
2. Copy your model to the model folder (or directly in N6_script folder)
3. Run the generate command:
   stedgeai generate -m your_model --target stm32n6 --st-neural-art 
4. Run the n6_loader.py script with DK board plugged in dev mode: 
   python n6_loader.py
5.  Run the validate on target command:
   stedgeai validate -m your_model --target stm32n6 --mode target -d serial:921600

Second method:

  1:go to stm32cubeMX,click the validate on target,and check the enabled,the click ok like this

 

 

The results is that I validate on target successfully by the first method,but get a problem by the second method.

I think the reason is that  config.json has been modified to  "cubeide_path": "".but i do not know what lead to this

phenomenon？

 

Thanks!

 

[SlothGrill]

Hello
Agreed, it's a bug on X-Cube-AI, then, as setting the path to "" will most likely lead to buggy flow.

I guess X-Cube-AI checks if cubeide has been installed on "standard" paths (i.e. not E:), and fallback to setting the config to "" when not found.

We will post a bug for that.

 

Thanks.

[cxf]

Hi @SlothGrill when I can get the latest X-Cube-AI which can solve this bug? As i want to validate on target by STM32CubeMX. Thanks very much!!!

[SlothGrill]

Hello,

X-Cube-AI gets updated when new versions of STEdgeAI are released. So there will be no available versions with a fix before any new STEdgeAI release (i.e. do not expect anything before some months...).

For now, if you really want to use X-Cube-AI, you can try to reinstall CubeIDE properly and see if it fixes the issues (eventually removing the old versions that may lie around on your PC may help). Or try to install it on standard target directories (eg. c:\ST\...)

Best regards.