STWINBX1 and HSDATALOG = Corrupt Data when converting to CSV

[WP_it]

I am running a STWINBX1 with the latest v1.2.0 firm from the STM32CubeFunctionPack_DATALOG2_V1.2.0.

I have compiled and uploaded the DATALOG2-STWIN.Box firmware successfully.  I am using the ST BLE Sensor App v5.0.0 on my iPhone to configure and run the "High Speed Data Log 2" option.  I configure the sensors (using the accelerometers only) and start and stop logging successfully.  

All good, but now I go to covert the .dat files to CSV files for processing using:

python hsdatalog_data_export.py d:\20230729_10_57_37 -f CSV -d

I am flooded with multiple lines like this in the terminal:

2023-07-29 11:00:52,572 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at -3.2819972584109675e+296 sec (HSDatalog_v2.py:564)
2023-07-29 11:00:52,573 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at -3.2819972584109675e+296 sec (HSDatalog_v2.py:564)

Or for another sensor

2023-07-29 10:54:47,672 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at nan sec (HSDatalog_v2.py:564)
2023-07-29 10:54:47,672 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at nan sec (HSDatalog_v2.py:564)
2023-07-29 10:54:47,672 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at nan sec (HSDatalog_v2.py:564)
2023-07-29 10:54:47,672 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at nan sec (HSDatalog_v2.py:564)
2023-07-29 10:54:47,672 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at nan sec (HSDatalog_v2.py:564)
2023-07-29 10:54:47,672 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at nan sec (HSDatalog_v2.py:564)
2023-07-29 10:54:47,673 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at nan sec (HSDatalog_v2.py:564)

Of course the data in the CSV file is corrupt, but mostly it looks like the Time column is the issue......the accelerometer data seems reasonable.

I am at a loss how the time stamp data can be so messed up.  I didn't have this issue with the previous version (v1.1) of the software but can't prove it.

Q:  Any ideas how to troubleshoot this?  No error messages, no debug data to diagnose.

Thank You

[RhSilicon]

Interesting to know that this data collector exists. I made a data collection project (Datalogger 2039) that also output a dat file and I also had to make a dat file to csv converter (also output excel file directly).

The issue of converting the dat file shouldn't be too complicated to get right if the file doesn't use encryption.

It may be possible to get help on the stackoverflow forums.

[SimonePradolini]

Hello @WP_it ,

can you share the entire acquisition folder or at least only the json files and the iis2dlpc_acc.dat?

By the log it seems that the acquisition is broken. Logging all the sensors at the maximum rate via SD card can fail if the SD card is not a class 10 one (we suggested some specific part number inside the User Manual). But we are not expecting this if you enabled only the IIS2DLPC .

Have you already done different acquisition with the same setup/configuration, by changing SD card or via USB?

Is the issue persisting also by enabling other sensors? Only IIS2DLPC is failing?

 

Best regards

Simone

[WP_it]

Here is the entire folder that will not convert without error as an example (and I put a copy of the corrupt CSV for reference).  I'll try with another SD card as I can't say I've tried that.  I did get one successful datalog and conversion done.  I didn't use the iPhone add, rather just used the USR button to start and stop the recording and it converted successfully.  Only tried that once so far so too early to say it always work or whether I just got lucky.

Appreciate the help!

 

[SimonePradolini]

Hello @WP_it ,

Sorry for the late reply. I've just analyzed your data: a timestamp is missing, so breaking the file.

Have you already tested your setup with another SD card and with different sensors configurations? I'm using your device_config.json with one of the suggested SD card and I'm generating working logs. The same configuration is working as expected also with acquisition in Python via USB.

 

Best regards

Simone

[WP_it]

Sorry for the slow reply as I've been away for a few days. 

Thank you very much for the help and I will wait for the BLESensor update in the coming days and report back.

Greatly Appreciated

[WP_it]

Only observation to share quickly is that it seems to work (record and convert to CSV) pressing the USR button on the STWINBX1 to start and stop, however now the folders are named DL2_00001, DL2_00002, etc instead of the nice date/time stamped versions before (eg: 20200811_09_09_22).  I will keep looking but all I've done so far is download v1.2.1 and build the DATALOG2 application for the STWIN.Box.

[SimonePradolini]

Hi @WP_it 

Just a brief comment. data+timestamp are sent by the BLE app or by the host PC (once started a data collection via USB). If there is no connection, the board cannot know the absolute time and date. That's why folder has only a sequential name.

This is the expected folder name format (already used also in previous DATALOG2 versions).

 

Best regards

Simone

[nicolas_tudarmstadt]

Hello,

I think I am having a similar issue. I am not using the app and SD card but I am recording the data via USB. The timestamps also get corrupted, especially in measurements that are longer than a few minutes, with the same errors / warnings that you got when trying to convert to .csv. I observe this behavior even in the HS GUI application sometimes. I am using the 1.2.1 version. Any help is much appreciated.

[WP_it]

I will give the USB option a try as I haven't used it for long durations....most of my testing has been relatively short.

Another question is how many sensors and at what rate are they recording?  For comparison most of my testing have been with only 2 sensors running at max 800Hz.  Just looking to compare/contrast to find out what's going on.

[nicolas_tudarmstadt]

Thanks!! At the moment I am using mainly the IIS3DWB but its recording at 27Khz, so quite a high datarate.

[WP_it]

I tried running in USB mode with the IIS3DWB running at max 27kHz and the IIS2DLPC and ISM330DHCX both running at about 200Hz.  I ran it for a a couple minutes and then converted all the files.  Good news is that it converted them all with no issue and no errors, bad news is that it's not very helpful for solving your issue.  I will try to run again with a "cheap & slow" SD card to see if that may trigger the issue.  

Stay tuned.

[SimonePradolini]

Just for your knowledge:

the issue from @nicolas_tudarmstadt is discussed also here and here

@WP_it can you confirm that your issue is solved?

 

Best regards

Simone

[WP_it]

Yes, I'm good now...problem resolved.

Thank You