How do I synchronize received UART data ?

[johngj]

I am trying to receive a frame containing 10 bytes of serial data, each frame is transmitted at a 100ms rate.

However, as the UART can start receiving the 10 byte frame at any point in time, the received data is becoming out of sync. 

How do I sync the UART peripheral so that it always starts receiving at the start of the 10 byte frame ?

Below is a scope capture of the 10 bytes of serial data:

Below is a scope capture of the 10 bytes of serial data being sent at a 100ms rate:

I am using the STM32 Nucleo-64 development board with STM32L433RC MCU. 

Its as if I need to monitor the serial line with a timer, in order to detect the period when the serial line is high for say 50ms and use this timer to enable and disable the UART peripheral.  But there must be a better way to do this ?

The UART is configured as shown below and the rx buffer is a size of 10.

The 10 bytes are transmitted by a electronic speed controller, used for radio control applications and contains the following data:

Byte 0: Temperature
Byte 1: Voltage high byte
Byte 2: Voltage low byte
Byte 3: Current high byte
Byte 4: Current low byte
Byte 5: Consumption high byte
Byte 6: Consumption low byte
Byte 7: Rpm high byte
Byte 8: Rpm low byte
Byte 9: 8-bit CRC

 

 

 

 

[Tesla DeLorean]

Typically people would have a preamble/sync byte, if you're in-sync try to receive the whole packet, if not receive one byte at a time until you get the sync byte, then request packet size minus one?

Most of the STM32 interrupt for every byte, it's only the HAL doing the collection and callback stuff.

You could perhaps use 9-bit mode, and use the 9th bit to indicate the first byte.

[Andrew Neil]

Lots of ways to do this

• Define a transmission format with a specified "start of frame" marker
• Define it by timing; eg, "1st byte after Xms quiet is start of frame"
• A query-response protocol
• A separate hardware sync signal
• etc, etc, ...

[KMill]

I'm 90% sure that MCU has IDLE time detection on the UART. Which means you can trigger an interrupt after a programmable amount if idle time on the RX line. Have a look at HAL_UARTEx_ReceiveToIdle_IT() etc

[Karl Yamashita]

Show your code so we can see how you're receiving the packet.

[tjaekel]

You are dealing with an asynchronous nature of this UART reception: yes, if you start "listening" (receiving) at any time, when the transmitter sends all the packets with 10 bytes - I can start right in the middle of a "packet".

I have similar issue as: I send a command via UART and the response can have any length (e.g. a long help text or a very short value).
If you run all characters in ASCII mode, means: all your bytes are characters in the hex value range 0x20...0x7F - you can use a special character, e.g. "End Of Text", coded as 0x03 byte: every "packet" (actually a string) ends with 0x03:

Wait for the 0x03 and know that this was the "End of Text" (end of package): check the length of received bytes: if it matches with 10 - fine, a complete package. If length is shorter: you have started receiving right in the middle of a packet:
discard this incomplete packet (which should happen only at the very first one, when you start to "synchronize" with the transmitter, afterwards you should be in sync).

If you run all in binary mode, means: all your bytes can have values in range 0x00..0xFF - you cannot use special control characters.

But you have the option to start the package with a length field, e.g. a byte which tells you how many bytes should follow.

If this mismatches - discard all and just wait for the next packet.

BTW:
An easier way, instead to "hook on" a asynchronous transfer, is to implement a handshaking: send the transmitter something and then it will respond: the transmitter side just sends something back when it was requested to do so.
This makes sure you are always in sync.

If you want to keep going with the approach, that you try to "listen" to a stream of packets, coming without any request ("ping-pong", no handshake), sent by the remote transmitter periodically, you can do this:

• as mentioned by Tesla DeLorean: have a prefix (preamble) and postfix (end character) - it marks you packet
  with START and END - but needs special control characters - you cannot use full binary packets
• receive at least two packets (the doubled packet size)
• and look for prefix and end character in this "byte stream"
• now you should be in sync
• you will find an "offset" where the prefix byte is located: all the next transmissions should have the same offset to the "start of packet" and you can keep going to take the correct entire packet from the input "stream buffer"

Remark:
Using a timer to measure if there was a gap between packets - it could work as well.
But is very dependent on packet size, the baud rate or if the transmitter would "stall" a bit, e.g. a gap between bytes because the host is busy to keep going with sending bytes in a sequence.

A handshake, at least you "release" the transmitter to start sending regularly the packets is the best approach.

Or use special characters to get in sync with the transmitter.

Or have a length field and use the length info to verify if packet received (in buffer) has the correct length.

Synchronizing with an asynchronous stream of packets needs a mechanism to "find" the packet in a stream and than to stay in sync with it. Once you have synchronized (you might need some packets received but place all in a "stream buffer") you should be fine from then on forward.

 

[johngj]

Thanks all

Unfortunately I have no control over the serial data that is being transmitted, so I cannot change the data frame or add handshaking etc

The serial data is a standard known as "KISS ESC telemetry" and is transmitted by an Electronic Speed Controller (ESC).  The ESC I am using is as follows...

https://powerdrives.net/120f3 

Details about the KISS ESC telemetry are explained in the attached pdf

I was hoping there might be a feature suggested by KMill i.e. an IDLE time detection on the UART so I can trigger an interrupt after a programmable amount if idle time on the RX line.  So I will investigate to see whether this is possible. 

 

 

[Andrew Neil]

---

@johngj wrote:

Details about the KISS ESC telemetry are explained in the attached pdf

---

So have you read that document?

It seems to tell you that you request the data when you want it:

"the telemetry is requested with a short PWM pulse of 30μS (+-2μS)"

So, once you've done that, you should expect to receive the 10 bytes.

That's what provides the synchronisation - it is, effectively, a request-response protocol.

It also tells you that there's a CRC - so you can use that to determine whether you have a valid "frame".

 

The document is short on details - you should contact Powerdrives for support:

https://powerdrives.net/contact 

 

[johngj]

Thanks Andrew,

Unfortunately I have no control over the telemetry request either.

The datalogger has to be passive and "listen in" to the telemetry serial data.

The telemetry request is performed by a flight controller, in this case a Pixhawk 6C mini and in this case using a protocol called DSHOT (for some reason RC seems to use non-standard protocols  :face_with_rolling_eyes:).

The flight controller sends the DSHOT message which requests the telemetry data.

My datalogger has to then listen for the telemetry data response (from the ESC) and store the data to an SD card.

I've already got the CRC check working in my code (which I attached in an earlier post - see sdcard.7z)

 

 

[johngj]

As requested by Karl Yamashita, I have attached my code.

There is also an SD card driver in my code, because I am trying to receive the serial data from the ESC and save that data to an SD card.

The project is to design a data logger that can be used with any ESC which uses the KISS telemetry.

 

[Karl Yamashita]

void HAL_UART_RxCpltCallback(UART_HandleTypeDef *huart)
{
 HAL_UART_Receive_DMA(&hlpuart1, rx_buff, 10); //You need to toggle a breakpoint on this line!
 process_telemetry();
}

I see you call process_telemetry but no where do i see that you are saving the data to the SD card? 

When you say the data is out of sync, do you mean that the packet contains some data from one packet and data from another packet? 

If somewhere you are saving the data to the SD card, are you sure you're not getting another interrupt before the data is saved? Thus, your latter part of the packet is being written over with new data?

Toggle an LED when it starts the process of saving the data and toggle again when it finishes saving the data. Then you can see if the toggle happens in between each UART packet.

[tjaekel]

Myself, I would think about these approaches to solve the issue:

1. receive a permanent stream and find the packet in receiver buffer:

Keep going to receive all what you see and place it in a buffer. The buffer should be at least twice the size of the packets, e.g. 20 bytes - or even longer, e.q. for Nx packet size (as 10 bytes). The buffer wraps to start (not a FIFO buffer with shifting inside the buffer).

When you have a finished DMA (assuming: using DMA and waiting for 10 bytes): iterate over the buffer and use the CRC check as indication where a packet seems to be correct:
Take 10 bytes from location 0, build CRC, if it mismatches: shift to start now to location 1 and do all again. Do this "incremental" move of packet start pointer until you find a correct CRC. Then the packet should be found.

Now you would know the "offset" where packets start in buffer. Keep going now to take packets with the "found offset".
(and bear in mind when the buffer wraps to beginning because buffer was full).

If the offset was found once - it should be clear where the packet starts are.
But you have to do all the time again when you do not keep going with this process, or you have a lost packet (start over with this "synchronization").

See David Littel's comment

2. Use the gap (time elapsed) to sync:

When you have for sure a difference between the gap the bytes of a packet are sent and the duration where nothing is sent - use this as indication where a new packet comes in:

Use UART in INT mode (not DMA), so, for every received byte you get an interrupt. This stores the bytes received in a buffer.

Measure now the elapsed time (e.g. via HAL_GetTick() ). If you see now that the next following byte comes 100 ms later (as the bytes before) - check the length received when you have started. Count on every INT the number of bytes you got and remember the time stamp.

If you see now, there was a gap (larger time now between two bytes) but you have not counted up yet to 10 (bytes received): the packet which you have in memory is incomplete: discard it and start again: back to buffer start, back to INT counter 0 and do again: receive and count, measure difference on time stamps. You should get now 10 bytes in a row before you see again "a gap" on time stamps. These 10 bytes are now your valid packet.

It works only in INT mode, with "single-byte" reception and measuring on every byte received the time (comparing time stamps). Not possible in DMA mode!

3. Monitor the external "sync" signal:

As I understood from previous comments: there is a signal which triggers the device to send these 10 bytes.
Why you cannot use this signal as "packet start" indication?
Configure a GPIO with INT (raising or falling edge), connect this signal as input to your MCU. When the two external devices are "talking" now to each other, you monitor this signal. You get a GPIO INT (with an edge) and this is an indication for you as: "now, make the UART Rx hot, be prepared to listen and to store the next 10 bytes in a buffer".

Now you should be in sync with the byte stream and you have locked onto the packets.

 

[David Littell]

You don't have to bother with synchronization, just save everything you receive to the telemetry file.  Whatever processes that file later can use the CRC to delineate the 10-byte blocks.  Any telemetry processing has to check the CRC anyway so use that requirement to simplify your recording code.

An efficient buffering scheme depends on how helpful your UART DMA is (a 20-byte circular with half-complete and complete interrupts is what you really want) and your filesystem's preferred blocking.

[Tesla DeLorean]

This, perhaps buffering with much larger blocks that are sector/cluster sized/aligned

There's also a COTS implementation using the STM32F411, these can be reprogrammed with custom code, but basically fire-hoses data from the serial port onto files on the MicroSD card.

https://github.com/d-ronin/openlager

https://www.aliexpress.com/i/3256804756627458.html?gatewayAdapt=4itemAdapt

 

[johngj]

I downloaded openlager from git but have no idea how you use it in Cube IDE.

If I 'Open projects from file system....' and point to the openlagger folder, everything opens but its not possible to build.

If I create a new STM32 project and add the folder to the project then I get lots of build errors

How do I use openlager in Cube IDE for the NUCLEO-L433RC-P ?