How does Ethernet RX descriptor wrapping work on STM32H7?

[PWint]

My Ethernet driver is not wrapping as I expect.

Section "61.10.1 Descriptor Overview" in RM0399 states that descriptors up to N-1 is owned by the DMA and the DMA continues to process descriptors until the Current Descriptor Pointer == Tail Pointer.

In order to exit suspend state, the tail pointer must be updated so that the Current Descriptor Pointer < Tail Pointer.

Using the example in Figure 841, the tail pointer points to entry 10 (which is not a valid entry since only 0 to 9 are valid). I can see that the DMA runs until the current descriptor points to 10, but it does not wrap back to zero. If the tail pointer must be larger than the current pointer, what should it be? Should it point to entry 11??

When will the DMA wrap, and start operating on buffer 0 again?

[Piranha]

As I know Ethernet well, but haven't worked on H7, I could be wrong, but from a quick look the whole page 3036 looks terribly misleading. It seems to describe not how a DMA hardware works, but how some limited mind thinks a driver should be implemented. And it seems that the ST's non-working H7 ETH HAL driver is implemented exactly like that... @alister​​, can you comment on this?

Meanwhile read alister's report:

https://community.st.com/s/question/0D50X0000C6eNNSSQ2/bug-fixes-stm32h7-ethernet

And take a look on other related issues:

https://community.st.com/s/question/0D50X0000BOtfhnSQB/how-to-make-ethernet-and-lwip-working-on-stm32

[PWint]

I agree, it does not seem to describe the DMA itself and implementing it as such does not work. Some issues I find:

1. The Descriptor ring length register (ETH_DMACRXRLR) must hod the number of descriptors. However, if I write it with the value 15 (meaning it must only access from index 0 to 14), then I can see the DMA accesses 16 buffers. I've set-up 16 descriptors and can see that descriptor at index 15 (the 16-th descriptor) is accessed. In order for my driver to work, I configure N buffers, but write the length register to N-1. The DMA in this case uses all N buffers.
2. I cannot figure out the purpose of the tail pointer. I am now writing the value zero to it every chance I get to keep the DMA going. It wraps correctly given my first point.
3. TX buffer seems to work differently. I cannot get TX running smoothly using same scheme. I will set-up a TX buffer, but it will not get transmitted until the second buffer gets written. I assume it has something to do with the TX DMA that is stalled, but cannot figure out how to fix it.

I have an in-house IP stack that I've ported to new hardware based on STM32H7 and is trying to write driver that plugs in to our IP stack. Never have I struggled this much getting Ethernet driver going and given my points above, I am totally confused as to what to do. I am in total trail-and-error mode which is not how this should work.

[alister]

>The Descriptor ring length register (ETH_DMACRXRLR) must hod the number of descriptors

It should be configured 1 less than your number of descriptors.

>I cannot figure out the purpose of the tail pointer

Refer some notes below.

>I will set-up a TX buffer, but it will not get transmitted until the second buffer gets written

Could only guess. Each descriptor may link two buffers. I suggest read the working driver at https://community.st.com/s/question/0D50X0000C6eNNSSQ2/bug-fixes-stm32h7-ethernet.

General DMA descriptor notes

1. ETH_DMACTXDLAR and ETH_DMACTXRLR are the tx DMA descriptor list start address and length respectively. These alone define the start and length of memory assigned to tx DMA descriptors. ETH_DMACRXDLAR and ETH_DMACRXRLR are the same for the rx DMA descriptors.
2. The ethernet DMA controller's current (internal) descriptor pointers are reset to ETH_DMACTXDLAR and ETH_DMACRXDLAR when ETH_DMACTXCR.ST and ETH_DMACRXCR.SR are set respectively.
3. The ethernet DMA controller reads the descriptors to determine where to read tx data and where to write rx data. Briefly, prior to starting receive or transmit, the descriptors must be initialized by writing the pointers to the receive or transmit buffers and the descriptor must be marked ready. For receive, the buffer size is configured by ETH_DMACRXCR.SBSZ. A descriptor is closed when frame is completely received or the buffer has reached ETH_DMACRXCR.SBSZ. A single receive frame may span one or more consecutive buffers. For transmit, each descriptor is configured its buffer's length.
4. The controller DMA may be running or suspended. While running, after reading a descriptor it marks it done, moves to the next, or from the last to the first when it reaches the list end, and continues until the current descriptor increments to a descriptor that is not ready or to the list tail pointer, ETH_DMACTXDTPR or ETH_DMACRXDTPR. The DMA will not start if the tail is written the current pointer.
5. For transmit, the controller is typically suspended, one or more descriptor are prepared and marked ready and then the DMA is started by writing ETH_DMACTXDTPR. Writing ETH_DMACTXDTPR is described as a "poll demand" command. A typical driver would free transmit buffers following a transmit-complete interrupt. 
6. For receive, a driver might be implemented two ways: (1) write ETH_DMACRXDTPR the address of the last descriptor at start-up or the most recently used descriptor after receiving a packet or (2) write ETH_DMACRXDTPR any other address so the current descriptor pointer never increments to it and receive continues until the current pointer increments to a non-ready descriptor. A typical driver would process the receive buffers on a receive-complete interrupt, assemble complete packets and feed those to its IP stack. The method (2) driver must handle the RBU (receive buffer unavailable) interrupt too for the non-ready descriptor case, perhaps caused by buffers being exhausted. If there are no buffers it would defer re-starting DMA until buffers have been freed.

[PWint]

Thanks, that helped a lot. Knowing what is right and what is not eliminated enough issues that I could get my driver going. Thanks also for your bug-fix in the ST driver. That, with your documentation was very useful.

My TX error turned out to be something else: I still have cache disabled (always start that way to get things going) but if I write the OWN bit in the TX descriptor, and then the tail pointer register to get DMA going, it would not see the change to the OWN bit yet. I had to add a memory barrier there to resolve my TX issue.

This makes sense since the ARM can do out-of-order writes with cache on or off.

[alister]

The cache cleaning and barrier for the descriptors could never out-perform using MPU region(s) for them.

[Piranha]

The following error report is for all reference manuals of H7 with ETH peripheral:

• RM0399 Rev 3 section 61.10.2
• RM0433 Rev 7 section 58.10.2
• RM0468 Rev 3 section 63.10.2

According to the information from others and common sense, the following bold formatted sentences in the respective sections are wrong and misleading:

---------

The Channel Tx descriptor tail pointer register (ETH_DMACTXDTPR) or Channel Rx descriptor tail pointer register (ETH_DMACRXDTPR) contains the pointer to the descriptor address (N). The base address and the current descriptor pointer decide the address of the current descriptor that the DMA can process. The descriptors up to one location less than the one indicated by the descriptor tail pointer (N – 1) are owned by the DMA. The DMA continues to process the descriptors until the following condition occurs:

Current Descriptor Pointer == Descriptor Tail Pointer;

The DMA enters the Suspend state when this condition occurs. The application must perform a write operation to the Descriptor tail pointer register and update the tail pointer so that the following condition is met:

Current Descriptor Pointer < Descriptor Tail Pointer;

---------

Descriptor tail pointer registers doesn't somehow magically "contain" some values and address of the descriptor N is not the best value for initialization also. Typically the best will be N+1 or anything that effectively disables stopping on tail pointer match. It is safe because even, if the DMA will fill all of the descriptors before those are processed by application and wrap around, it will still enter the suspended state, because the OWN bit on the descriptor 0 will still be cleared. Therefore the tail pointer does not determine the ownership of descriptors and the text is misleading. And relational operator "<" is not the logical opposite of "==". The "!=" is, which includes also the ">" case. Common sense and point 4 in Alister's detailed comment above tells that hardware just stops processing descriptors, if the tail pointer equals current descriptor address, and doesn't do "less than" comparison, because otherwise the feature would be practically useless.

And there is another tail pointer related misinformation in register descriptions:

---------

Channel Tx descriptor tail pointer register (ETH_DMACTXDTPR)

The ChannelTx Descriptor Tail Pointer register points to an offset from the base and indicates the location of the last valid descriptor.

Channel Rx descriptor tail pointer register (ETH_DMACRXDTPR)

The Channel Rx Descriptor Tail Pointer Points to an offset from the base and indicates the location of the last valid descriptor.

---------

Besides the missing space in "ChannelTx", missing word "register" for Rx and an unnecessary capital letter in "Points", the information says those registers hold the offset from the descriptor list base address. Again an information from others and even the HAL ETH driver on the contrary shows that those registers hold the absolute address of the tail pointer.

And one more thing. Contrary to the reference manuals, CMSIS header files name ETH registers (not only these two) without the "X" letter (formatted as bold in the above example) in the names. One or the other must be corrected.

Other topics confirming the presented information:

https://community.st.com/s/question/0D50X0000C6eNNSSQ2/bug-fixes-stm32h7-ethernet

https://community.st.com/s/question/0D53W000003vOM1SAM/stm32h7-ethernet-please-help-to-review-rx-descriptor-init-fix

https://community.st.com/s/question/0D50X0000AU4PuySQF/ethernet-receive-stability-using-tcp

@Imen DAHMEN​, this information should be confirmed and the reference manuals corrected.

[Imen.D]

Hello @Piranha​ ,

Thank you for pointing out the errors.

I escalated your feedback internally for review and correction.

Thanks

Imen

[Pavel A.]

@Imen DAHMEN​ ,

Thank you for the care.

By this occasion, could you check another point?

It seems that addresses of RX buffers must be aligned on 512 bytes.

In my testing, lesser alignment (on 32 bytes for example) caused errors in RX data.

The RM does not mention this.

Also, the description text on BUF1AP, BUF2AP in RX descriptors has something about LS bits, shifts, offsets that looks confusing. RM0433 tables 540,542 on pg. 2866

Thanks,

-- pavel

[MWB_CHa]

Hello @Pavel A.​ ,

Sorry for the delayed answer.

We confirm this issue was present in the legacy ETH HAL driver (alignment on 512 bytes).

But this problem was fixed in the new ETH HAL and now Rx Buffers are allocated from LwIP stack memory heap, then passed to ETH HAL driver (__ALIGNED(32)).

Kind Regards,

[Pavel A.]

Dear Chekib, thank you for the update. Good to know.