Hardware IPv4 checksum on an STM32F407 is not working, though all the proper settings are set. (Works on rare occasions, oddly)

[SHers]

I'm debugging a legacy application on an STM32F407 that uses the std peripheral libraries and LwIP 1.4.1, and except on rare occasions, hardware IPv4 checksums don't work; they all get sent out as zeroes. When I disable hardware checksums in lwipopts.h, I get correct checksums (but at a performance penalty). The application works reliably in all other respects, and we've previously ignored the checksum issue.

My development IDE is Keil MDK 5.36.0.0

Ethernet link is 100BaseT, fixed IPv4 addressing (no DHCP), with two different PHY chips on different target systems (both systems use the same Ethernet stack code and exhibit the same bug).

There are a number of settings that need to be used for this, and after many searches, much reading, and a number of debug sessions, I'm confident I have set them all to the right settings. (I've gone as far as breakpointing the Ethernet transmission when it is about to hand over the Ethernet DMA TX descriptor, and checking the descriptor fields and Ethernet registers at that time.)

• CHECKSUM_BY_HARDWARE defined in lwipopts.h
• USE_ENHANCED_DMA_DESCRIPTORS defined in stm32f4x7_eth_conf.h
• Extended Tx descriptors are used, and are initialized with CIC bits set to ETH_DMATxDesc_CIC_TCPUDPICMP_Full
• ETH_DMAMBR EDFE bit set to 1
• ETH_MACCR IPCO bit set to 1
• IPv4 ethertype used for transmit

The fact that I have had ONE debug session that did return nonzero checksums with hardware calculation enabled leads me to suspect that I'm configuring the registers and descriptors correctly, but some subtle interference is happening. I'm aware of the IPv6 erratum, but I don't think it applies here.

Where else should I look for the cause of this?

Thanks,

Steve Hersey

[waclawek.jan]

Try to review the Transmit checksum offload section in ETH chapter of RM. In particular, it talks about ETH_DMAOMR.TSF bit, and perhaps more. You should also check the status bits in the descriptor after the ETH releases it.

JW

[SHers]

Thanks for the suggestions.

Yes, that section of the RM would be dog-eared by now if it were a hardcopy. I've read it several times, searched the RM for every instance of "offload" to be sure I didn't miss anything, and done the same with the errata sheet in case that is relevant.

I forgot to mention that I am indeed setting the TSF bit (and the debugger assures me it is staying set). That's a good catch, though.

The post-transmission status of the descriptor is a good idea, thanks. ISTR that the MAC will set error bits there if it can't properly manage the header checksum calculation.

Alas, those error bits aren't being set. After transmission of the ICMP echo response, the descriptor status is 0x30D00000, which seems to indicate that all is well there.

I've also checked the clock tree and enabled the CRC module clocks and Ethernet PTP clock, in case those were at fault. Everything seems to be as it should.

This is a weird one and no mistake.

[SHers]

Aaaand, solved!

This one is a combination of legacy code doing unexpected things, and hardware checksum calculation having unexpected requirements.

1) The LwIP 1.4.1 stack has an entirely SEPARATE set of #defines in opt.h that are supposed to be controlled by a single CHECKSUM_BY_HARDWARE #define in lwipopts.h. For example, defining CHECKSUM_BY_HARDWARE sets CHECKSUM_GEN_IP, etc, to 0 for hardware checksum generation. BUT. Someone didn't define CHECKSUM_GEN_ICMP in that block with the rest, so ICMP checksums were generated in software anyway. Probably an LwIP bug, but it's hard to know for certain, and 1.4.1 has been superseded anyway.

And 2) The STM32F4 Ethernet peripheral is picky; it wants to initially see zeroes in the checksum fields it is going to hardware-calculate. If the checksum field goes in zero, it comes out on the wire calculated correctly. But if it goes in nonzero, it comes out on the wire as zeroes instead.

Solution: #defining CHECKSUM_GEN_ICMP as 0 in lwipopts.h makes the transmitted ICMP packets magically have proper hardware checksums. So make sure those checksum fields are zeroed first if you want the Ethernet MAC to hardware-calculate them.

[waclawek.jan]

Thanks for coming back with the solution. This is a gotcha indeed.

Please select your post as Best so that the thread is marked as solved.

I have a faint recollection of what you say, but can't find any direct pointers. Contrary, RM0090 says in the mentioned section:

 The input frame’s checksum field is ignored during calculation and replaced by the calculated value.

I am not surprised. The Synopsys modules (ETH and OTG) are notoriously badly documented.

(@Amel NASRI​ , can this please be reviewed/fixed?)

My faint recollection made me to review some older works. This is part of a bootloader I wrote barebones, without relying on lwip or other library, although certainly starting from the lwip implementation. This snippet takes the incoming ETH frame, parses it, and if it's ICMP, generates the reply in place. Note, that the checksum field is explicitly zeroed - this may be simply a coincidence, or I might've stumbled upon the same thing as you and just corrected it and forgot about it [EDIT] Or, most probably, I might've read this thread or a similar one, I was following the lwip forum back then [/EDIT]

{
 TIpHeader * iphdr;
 uint32_t ipHdrLen;
 uint32_t ipLen;
 uint8_t * pIpData; // to be passed to upper protocols
 uint32_t ipDataLen; // ditto
 
 if (len < (SIZEOF_ETH_HDR + sizeof(TIpHeader))) break; // drop too short packet
 iphdr = (TIpHeader *)(&ethhdr->data);
 if (iphdr->ver != 4) break; // drop non-IPV4
 ipHdrLen = iphdr->hdrLen * 4;
 ipLen = swap16(iphdr->len);
 if (ipHdrLen < 20) break; // invalid len
 if (ipHdrLen > len - SIZEOF_ETH_HDR) break; // too big header, not enough data
 if (ipLen > len - SIZEOF_ETH_HDR) break; // too big IP packet, not enough data
 pIpData = ((uint8_t *)iphdr) + ipHdrLen;
 ipDataLen = ipLen - ipHdrLen;
 if ((iphdr->offset AND IP_MF) != 0) break; // absolutely no support for fragmenting, RFC or not
 #ifndef CHECKSUM_BY_HARDWARE
 #error "CHECKSUM_CHECK_IP not implemented, as CHECKSUM_BY_HARDWARE assumed"
 #endif
 if (!((iphdr->dst[0] == ipAddr[0]) ANL (iphdr->dst[1] == ipAddr[1]) ANL (iphdr->dst[2] == ipAddr[2]) ANL (iphdr->dst[3] == ipAddr[3]))) {
 break; // reject if not for us (we should reject eth-broadcasts here too, btw)
 }
 switch (iphdr->protocol) {
 default: break;
 case 1: // ICMP
 
 {
 TIcmpHeader * icmpHdr;
 if (ipDataLen < sizeof(TIcmpHeader)) break; // drop too short packets
 icmpHdr = (TIcmpHeader *)pIpData;
 if (icmpHdr->type != 8) break; // respond only to pings
 if (icmpHdr->code != 0) break;
 // we rely on CHECKSUM_BY_HARDWARE to check ICMP checksum, too
 // we recycle the packet
 icmpHdr->type = 0; // echo response
 icmpHdr->checksum = 0; // again we rely on CHECKSUM_BY_HARDWARE to generate ICMP checksum, too
 iphdr->checksum = 0; // similarly with IP
 iphdr->ttl = 255;
 memcpy(iphdr->dst, iphdr->src, sizeof(iphdr->src));
 memcpy(iphdr->src, ipAddr, sizeof(ipAddr));
 memcpy(ethhdr->dest, ethhdr->src, sizeof(ethhdr->src));
 memcpy(ethhdr->src, macAddr, sizeof(macAddr));
 EthSendPacket(ethhdr, ipLen + SIZEOF_ETH_HDR); // sending all received length ensures that data is returned untouched
 }

JW

[SHers]

Thanks for the additional information. It's ironic that the LwIP thread you linked to is exactly the information I was trying to ferret out by the use of search-engine-foo (but never quite reached). There's a lot of information out there, often the hard part is "asking" the right "question" in the right place (and hoping the answer was ever indexed and shows up ahead of the pages of cruft and crap)...

And I can testify from direct experience that RM 0090 is in error there. Needs an erratum added, as you pointed out.

Regards,

Steve