receive unknown length frames with UART

[Adam98]

Hello everyone,

I’m working on an embedded project where I need to receive a variable-length stream of bytes via UART on an MCU  STM32H750. The key points are:

• All bytes are payload, no start or stop bytes, no protocol framing.

• The number of bytes is undefined and variable.

• The byte stream starts when the MCU is ready.

• I need to calculate CRC (Ethernet CRC32) over all received bytes.

• After receiving all data, I must output the CRC via UART.

• The UART baud rate is very high (greater than 900,000 baud).

My current approach:

• I use UART interrupts to receive bytes one by one.

• I update the CRC continuously with every received byte inside the interrupt handler.

• This works fine for small data, and CRC calculation seems correct.

• However, when I try to receive a large file or big data stream, the CRC isnt correct, the crc will be outputed before the whole data recieved.

I suspect the problem is related to handling the continuous stream without any start/stop or length information, and possibly buffer management or interrupt handling at such a high baud rate.

My questions:

1. How can I reliably receive an undefined-length UART stream without framing or start/stop bytes?

2. What techniques can I use to detect the end of transmission if there is no protocol?

3. Are there best practices for handling UART reception and CRC calculation in an interrupt-driven system at very high baud rates?

4. How can I avoid losing bytes or corruption when the incoming data size is large?

5. Any example code snippets or references for a similar use case would be great!

6. I try to use dma (ping pong buffer and update crc but it dosent work because i cant find the best time and place to calculate crc...)

Thanks a lot for your help!

[TDK]

Use HAL_UARTEx_ReceiveToIdle_DMA and use the IDLE event to know when a frame stops.

STM32CubeF4/Projects/STM32446E-Nucleo/Examples/UART/UART_ReceptionToIdle_CircularDMA/Src/main.c at 0c04cc8f2360e54ced149afb43195a03f529a1c5 · STMicroelectronics/STM32CubeF4

[Pavel A.]

> After receiving all data, I must output the CRC via UART.

How do you decide that all data is received?  

Do you need to calculate the CRC over *all* data or over each "stream" or how you define the "units" of the data?

STM32H7 can handle continuous UART RX interrupts at ~ 100 per ms, with properly written code (not the "HAL" library).  Enabling the UART FIFO can help as well. Using the CRC accelerator of STM32 can help, if the CRC algorithm is suitable for you.  DMA [with timeout/idle detection] seems more attractive, but more complicated.

 

> How can I avoid losing bytes or corruption when the incoming data size is large?

An excellent question. The "streams" should not be too long, exactly with purpose of lessening chance of loss or corruption within single "stream" and minimize retransmissions.  This cannot be avoided and should be handled properly.

 

 

[Karl Yamashita]

To get a better understanding of what is going on...

• If the packet has no protocol or some kind of identifier, how do you know which packet is what?
• What are the purpose of these variable length packets after you calculate the CRC and send it out over UART?
• Is it possible that the sending device may pause in the middle of sending the large packet?
• How often are the packets being sent?
• What are some examples size of these large packets?

 

My questions:

1. How can I reliably receive an undefined-length UART stream without framing or start/stop bytes? Use DMA

2. What techniques can I use to detect the end of transmission if there is no protocol? Use DMA with idle.

3. Are there best practices for handling UART reception and CRC calculation in an interrupt-driven system at very high baud rates? Don't calculate in the interrupt/callback

4. How can I avoid losing bytes or corruption when the incoming data size is large? Use DMA

5. Any example code snippets or references for a similar use case would be great! 

6. I try to use dma (ping pong buffer and update crc but it dosent work because i cant find the best time and place to calculate crc...) You need to save the DMA data to a queue buffer as the TC and HT occur. When you get an idle interrupt, you can increment the queue pointer. So as you're calculating the CRC on the 1st queue in the main while loop, any new DMA data will be saved to the next queue. That way you don't miss any incoming data as you're incrementing through the current queue and calculating the CRC

---

 

[Adam98]

thank you alll

The Programm is now working using HAL_UARTEx_ReceiveToIdle_DMA and use the IDLE event to know when a frame stops.

 

but another problem that my programm works only with baud rates until 460800. if i want to use 921600 baud rates it dosent work properly and the value of the CRC is wrong!!!

[Tesla DeLorean]

Perhaps look at WHY the CRC is wrong..

Which data bytes did you receive that were incorrect or missing?

What's the aversion to any kind of protocol or framing? Something like SLIP is a very simple method to identify beginning and ending frames, whilst still permitting any 8-bit byte value to get from one end to the other.

Other protocols, perhaps use preambles and lengths. What about multi-controller / 9-bit methods for identifying data vs control bytes?

Using inter-symbol or inter-packet gaps that you can generate and detect effectively?

[Karl Yamashita]

Depending on your clock speed, setting for 921600 doesn't mean it's exactly 921600. If you play with the clock especially with a high baud rate, sometimes CubeMX/STM32CubeIDE will give a warning that there is an error greater than some %.

There could still be a slight % error, but CubeMX may not give a warning if it's under a threshold? I'm just assuming about the threshold, so take that with a grain of salt. 

Since you're using TeraTerm, i assume you're using some USB-Serial adapter. That adapter too, could have a % error.  

Maybe check with a scope to see if indeed both devices are exactly 921600?

[Pavel A.]

> There could still be a slight % error, but CubeMX may not give a warning if it's under a threshold? I'm just assuming about the threshold,

Why CubeMX should be concerned? 1Mbit/s is not a terribly high rate for UART (maybe 2.5 Mbit/s is marginal) but can work with proper connections. CubeMX is not responsible for this. For higher bitrates better ways exist, based on LVDS and similar.  Reasonable data corruptions can be worked around by software. (sorry, can't give any ready code to copy-paste).

 

 

[Karl Yamashita]

I must be getting too much caffeine and not seeing straight.

 

 

[Karl Yamashita]

I have a H750 to work with. Can you upload your IOC file so I can start off with your configuration?

[Adam98]

i have uploaded now