STM32MP1 ethernet issue

[stefboerrigter]

Hi,

We are using the STM32MP157CAA3 SoC in our design and are having some Ethernet issues:

As we had problems sourcing the Realtek PHY from the reference design we use the microchip LAN8830T-V/PSA Ethernet PHY which should support Gigabit Ethernet.
When setting booting the device and connecting an Ethernet cable we observe the following issue:
1. With 10 MBit or router, the device probes and from both U-boot and Linux we can do dhcp request. But we never get an answer (no IP etc). 
2. With 100 MBit or 1000 MBit speed, the linux kernel throws a panic after some time related to DMA (sometimes also the probing fails for the DMA engine).

Both paths never show any data coming through the Ethernet port:  (DCHP requests seem to be unanswered (don't arrive on server side) and ping also doesn't arrive. LED's remain stale (green = constant ON, orange LED = off). 

Bootlog during success:

root@board:~# dmesg | grep eth
[ 0.000000] psci: probing for conduit method from DT.
[ 0.269703] usbcore: registered new interface driver cdc_ether
[ 0.289476] optee: probing for conduit method.
[ 1.633140] stm32-dwmac 5800a000.ethernet: IRQ eth_lpi not found
[ 1.638473] stm32-dwmac 5800a000.ethernet: no regulator found
[ 1.638983] stm32-dwmac 5800a000.ethernet: User ID: 0x40, Synopsys ID: 0x42
[ 1.644693] stm32-dwmac 5800a000.ethernet: DWMAC4/5
[ 1.649759] stm32-dwmac 5800a000.ethernet: DMA HW capability register supported
[ 1.656999] stm32-dwmac 5800a000.ethernet: RX Checksum Offload Engine supported
[ 1.664289] stm32-dwmac 5800a000.ethernet: TX Checksum insertion supported
[ 1.671191] stm32-dwmac 5800a000.ethernet: Wake-Up On Lan supported
[ 1.677637] stm32-dwmac 5800a000.ethernet: TSO supported
[ 1.682709] stm32-dwmac 5800a000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[ 1.690628] stm32-dwmac 5800a000.ethernet: Enabled L3L4 Flow TC (entries=2)
[ 1.697535] stm32-dwmac 5800a000.ethernet: Enabled RFS Flow TC (entries=10)
[ 1.704522] stm32-dwmac 5800a000.ethernet: TSO feature enabled
[ 1.710332] stm32-dwmac 5800a000.ethernet: Using 32/32 bits DMA host/device width
[ 4.623237] stm32-dwmac 5800a000.ethernet eth0: Register MEM_TYPE_PAGE_POOL RxQ-0
[ 4.631424] stm32-dwmac 5800a000.ethernet eth0: PHY [stmmac-0:00] driver [Generic PHY] (irq=POLL)
[ 4.644235] stm32-dwmac 5800a000.ethernet eth0: No Safety Features support found
[ 4.651974] stm32-dwmac 5800a000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[ 4.660496] stm32-dwmac 5800a000.ethernet eth0: registered PTP clock
[ 4.668620] stm32-dwmac 5800a000.ethernet eth0: configuring for phy/rgmii-id link mode
[ 8.880230] stm32-dwmac 5800a000.ethernet eth0: Link is Up - 10Mbps/Full - flow control rx/tx
[ 8.887340] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Bootlog during failure:

root@wesl:~# dmesg | grep eth
[ 0.000000] psci: probing for conduit method from DT.
[ 0.269963] usbcore: registered new interface driver cdc_ether
[ 0.289696] optee: probing for conduit method.
[ 1.633444] stm32-dwmac 5800a000.ethernet: IRQ eth_lpi not found
[ 1.638790] stm32-dwmac 5800a000.ethernet: no regulator found
[ 1.639302] stm32-dwmac 5800a000.ethernet: User ID: 0x40, Synopsys ID: 0x42
[ 1.645012] stm32-dwmac 5800a000.ethernet: DWMAC4/5
[ 1.650074] stm32-dwmac 5800a000.ethernet: DMA HW capability register supported
[ 1.657319] stm32-dwmac 5800a000.ethernet: RX Checksum Offload Engine supported
[ 1.664610] stm32-dwmac 5800a000.ethernet: TX Checksum insertion supported
[ 1.671509] stm32-dwmac 5800a000.ethernet: Wake-Up On Lan supported
[ 1.677850] stm32-dwmac 5800a000.ethernet: TSO supported
[ 1.683023] stm32-dwmac 5800a000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[ 1.690841] stm32-dwmac 5800a000.ethernet: Enabled L3L4 Flow TC (entries=2)
[ 1.697848] stm32-dwmac 5800a000.ethernet: Enabled RFS Flow TC (entries=10)
[ 1.704736] stm32-dwmac 5800a000.ethernet: TSO feature enabled
[ 1.710643] stm32-dwmac 5800a000.ethernet: Using 32/32 bits DMA host/device width
[ 4.976410] stm32-dwmac 5800a000.ethernet eth0: Register MEM_TYPE_PAGE_POOL RxQ-0
[ 4.985026] stm32-dwmac 5800a000.ethernet eth0: PHY [stmmac-0:00] driver [Generic PHY] (irq=POLL)
[ 5.997554] stm32-dwmac 5800a000.ethernet: Failed to reset the dma
[ 6.002358] stm32-dwmac 5800a000.ethernet eth0: stmmac_hw_setup: DMA engine initialization failed
[ 6.011322] stm32-dwmac 5800a000.ethernet eth0: __stmmac_open: Hw setup failed
[ 9.224884] stm32-dwmac 5800a000.ethernet eth0: Register MEM_TYPE_PAGE_POOL RxQ-0
[ 9.241514] stm32-dwmac 5800a000.ethernet eth0: PHY [stmmac-0:00] driver [Generic PHY] (irq=POLL)
[ 9.254347] stm32-dwmac 5800a000.ethernet eth0: No Safety Features support found
[ 9.262215] stm32-dwmac 5800a000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[ 9.271172] stm32-dwmac 5800a000.ethernet eth0: registered PTP clock
[ 9.278538] stm32-dwmac 5800a000.ethernet eth0: configuring for phy/rgmii-id link mode
[ 13.439784] stm32-dwmac 5800a000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[ 13.446896] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 19.687950] NETDEV WATCHDOG: eth0 (stm32-dwmac): transmit queue 0 timed out
[ 19.839844] stm32-dwmac 5800a000.ethernet eth0: Reset adapter.
[ 19.855494] stm32-dwmac 5800a000.ethernet eth0: Register MEM_TYPE_PAGE_POOL RxQ-0
[ 19.863823] stm32-dwmac 5800a000.ethernet eth0: PHY [stmmac-0:00] driver [Generic PHY] (irq=POLL)
[ 19.876709] stm32-dwmac 5800a000.ethernet eth0: No Safety Features support found
[ 19.884033] stm32-dwmac 5800a000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[ 19.892921] stm32-dwmac 5800a000.ethernet eth0: registered PTP clock
[ 19.898791] stm32-dwmac 5800a000.ethernet eth0: configuring for phy/rgmii-id link mode
[ 24.079787] stm32-dwmac 5800a000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[ 24.086899] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Panic:
[ 19.676490] ------------[ cut here ]------------
[ 19.679713] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x208/0x27c
[ 19.687950] NETDEV WATCHDOG: eth0 (stm32-dwmac): transmit queue 0 timed out
[ 19.694933] Modules linked in:
[ 19.697987] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.1.28 #1
[ 19.703864] Hardware name: STM32 (Device Tree Support)
[ 19.709041] unwind_backtrace from show_stack+0x10/0x14
[ 19.714232] show_stack from dump_stack_lvl+0x40/0x4c
[ 19.719318] dump_stack_lvl from __warn+0x78/0x158
[ 19.724102] __warn from warn_slowpath_fmt+0x148/0x1c0
[ 19.729187] warn_slowpath_fmt from dev_watchdog+0x208/0x27c
[ 19.734882] dev_watchdog from call_timer_fn+0x34/0x1a8
[ 19.740079] call_timer_fn from __run_timers+0x274/0x378
[ 19.745370] __run_timers from run_timer_softirq+0x2c/0x40
[ 19.750863] run_timer_softirq from __do_softirq+0x138/0x408
[ 19.756555] __do_softirq from __irq_exit_rcu+0xc0/0x10c
[ 19.761837] __irq_exit_rcu from irq_exit+0x8/0x10
[ 19.766609] irq_exit from __irq_svc+0x88/0xc8
[ 19.771077] Exception stack(0xc1201ee8 to 0xc1201f30)
[ 19.776048] 1ee0: 00000000 00000000 0001c359 c0119740 00000000 c1208f40
[ 19.784247] 1f00: c1324320 c1208f40 00000000 00000000 10c5387d 00000000 7fffffff c1201f38
[ 19.792441] 1f20: c0109080 c0109084 60030013 ffffffff
[ 19.797504] __irq_svc from arch_cpu_idle+0x38/0x3c
[ 19.802287] arch_cpu_idle from default_idle_call+0x40/0x12c
[ 19.807977] default_idle_call from cpuidle_idle_call+0x18c/0x1ec
[ 19.814067] cpuidle_idle_call from do_idle+0x94/0xc8
[ 19.819145] do_idle from cpu_startup_entry+0x18/0x1c
[ 19.824122] cpu_startup_entry from rest_init+0xd0/0xd4
[ 19.829406] rest_init from arch_post_acpi_subsys_init+0x0/0x8
[ 19.835205] ---[ end trace 0000000000000000 ]---
[ 19.839844] stm32-dwmac 5800a000.ethernet eth0: Reset adapter.
[ 19.855494] stm32-dwmac 5800a000.ethernet eth0: Register MEM_TYPE_PAGE_POOL RxQ-0
[ 19.863823] stm32-dwmac 5800a000.ethernet eth0: PHY [stmmac-0:00] driver [Generic PHY] (irq=POLL)
[ 19.871457] dwmac4: Master AXI performs any burst length
[ 19.876709] stm32-dwmac 5800a000.ethernet eth0: No Safety Features support found

We have the idea that it is clock-tree related (See attached) .

The design is strapped so the PHY generates the clock-source (125MHz for 1GBit) 

Some matching issues that we searched on the forum: 

•  https://community.st.com/t5/stm32-mpus-embedded-software/stm32mp157-ethernet-problem/td-p/280709 pointed us to check the strapping to have the clock generated by the PHY

Relevant part of our schematics:

Our OPtee/tf-a rcc config see attached dts files (actually dtsi files, but not allowed). 
both are included by dts file like:

/dts-v1/;
#include "stm32mp157.dtsi"
#include "stm32mp15xc.dtsi"
#include "stm32mp15-pinctrl.dtsi"
#include "stm32mp15xxac-pinctrl.dtsi"
#include "stm32mp15xx-<ourBoard>-common.dtsi" //<-- the attached files
/ {

 model = "proto 0 board";
 compatible = "st,stm32mp157c-dk2", "st,stm32mp157";

 chosen {
 stdout-path = "serial0:115200n8";
 };
};

 

Clocks measured:

Xtal connected to PHY = 25Mhz,
Clock output from PHCLK125= 125Mhz,
RX Clock frequency = 25Mhz,
TX Clock frequency = 2,5Mhz. [Why different compared to RX?]

We have some design length differences in signals from Ethernet controller to PHY, which are a in pico second range and should be within limit of the PHY:

 

Signal length of lines & calculated delays

Name	Signal Length (mm)	Delay (ps)
ETH1_GTX_CLK_PP1	133	910
ETH1_MDC_PP1	144	990
ETH1_MDIO_PP1	153	1052
ETH1_RXD0-NetR1003_2_PP1	137	935
ETH1_RXD1-NetR1002_2_PP1	143	974
ETH1_RXD2-NetR1001_2_PP1	121	799
ETH1_RXD3-NetR1000_2_PP1	162	1110
ETH1_RX_CLK-NetR1004_2_PP1	160	1098
ETH1_RX_CTL-NetR1005_2_PP1	126	833
ETH1_TXD0_PP1	188	1294
ETH1_TXD1_PP1	155	1038
ETH1_TXD2_PP1	181	1239
ETH1_TXD3_PP1	205	1400
ETH1_TX_CTL_PP1	121	808
ETH_CLK125OUT-NetR1006_2_PP1	155	1058
ETH_GPIO2-NetR1044_1_PP1	203	1407
ETH_GPIO3-NetR1037_1_PP1	169	1163
ETH_GPIO4-NetR1038_1_PP1	160	1101
ETH_GPIO5-NetR1039_1_PP1	131	867
ETH_RESET_N-NetC1033_2_PP1	144	982

 

[stefboerrigter]

Note; we have updated the table with the delays for better understandability:
Please note the column "Delay Datalines VS clock (ps)" for absolute delay in ps related to shortest path (RX lines compared to RXC and TX lines compared to TXC). 

Info	Name	Signal Length (mm)	Routed Length (mm)	Delay (ps)	Delay (ps)	Delay Datalines Delta (ps)	Delay Datalines VS clock (ps)	Delta length (mm)
RXD0	ETH1_RXD0-NetR1003_2_PP1	137.343	137.343	935.370	935,37	135,945	163	15.818
RXD1	ETH1_RXD1-NetR1002_2_PP1	143.291	143.291	974.614	974,614	175,189	124	21.766
RXD2	ETH1_RXD2-NetR1001_2_PP1	121.525	121.525	799.425	799,425	0	299	0
RXD3	ETH1_RXD3-NetR1000_2_PP1	162.812	162.812	1.110.454	1110,454	311,029	12	41.287
RXC	ETH1_RX_CLK-NetR1004_2_PP1	160.531	160.531	1.098.753	1098,753		0	39.006
RX_CTL	ETH1_RX_CTL-NetR1005_2_PP1	126.011	126.011	833.283	833,283			4.486
TX	Tx Deltas:
TXD0	ETH1_TXD0_PP1	188.738	188.738	1.294.866	1294,866	256,574	384	66.870
TXD1	ETH1_TXD1_PP1	155.307	155.307	1.038.292	1038,292	0	128	33.439
TXD2	ETH1_TXD2_PP1	181.008	181.008	1.239.061	1239,061	200,769	328	59.140
TXD3	ETH1_TXD3_PP1	205.391	205.391	1.400.498	1400,498	362,206	490	83.523
TXC	ETH1_GTX_CLK_PP1	133.627	133.627	910.679	910,679		0	11.759
TX_CTL	ETH1_TX_CTL_PP1	121.868	121.868	808.406	808,406			0

[stefboerrigter]

Update after some investigation
1. We have modified our board (reverted basically) to NOT use the 125MHz from the Phy. We have enabled in TF-A & OP-Tee the 

 

CLK_ETH_PLL4P

 

which is configured for 125Mhz via the HSE (24MHz) divmn = <3, 124>; st,pll_div_pqr=<5 7 7>; 

2. In Kernel DTS we enable the st,eth-clk-sel so taht in dwmac-stm32.c during 'stm32mp1_set_mode'  in RGMII mode the enable_eth_ck is activated. We have briefly checked with 25Mhz clock but that doesn't seem to work (get DMA issues reported in dmesg again).

(Relevant) Dmesg output:

 

root@wesl:~# dmesg | grep eth -i
[ 0.000000] psci: probing for conduit method from DT.
[ 0.269579] pegasus: Pegasus/Pegasus II USB Ethernet driver
[ 0.269869] usbcore: registered new interface driver cdc_ether
[ 0.289639] optee: probing for conduit method.
[ 1.633461] stm32-dwmac 5800a000.ethernet: IRQ eth_lpi not found
[ 1.638406] stm32-dwmac 5800a000.ethernet: SB: ETH Clock is selected
[ 1.644698] stm32-dwmac 5800a000.ethernet: no regulator found
[ 1.645213] stm32-dwmac 5800a000.ethernet: User ID: 0x40, Synopsys ID: 0x42
[ 1.651507] stm32-dwmac 5800a000.ethernet: 	DWMAC4/5
[ 1.656380] stm32-dwmac 5800a000.ethernet: DMA HW capability register supported
[ 1.663688] stm32-dwmac 5800a000.ethernet: RX Checksum Offload Engine supported
[ 1.670992] stm32-dwmac 5800a000.ethernet: TX Checksum insertion supported
[ 1.677790] stm32-dwmac 5800a000.ethernet: Wake-Up On Lan supported
[ 1.684208] stm32-dwmac 5800a000.ethernet: TSO supported
[ 1.689404] stm32-dwmac 5800a000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[ 1.697220] stm32-dwmac 5800a000.ethernet: Enabled L3L4 Flow TC (entries=2)
[ 1.704109] stm32-dwmac 5800a000.ethernet: Enabled RFS Flow TC (entries=10)
[ 1.711128] stm32-dwmac 5800a000.ethernet: TSO feature enabled
[ 1.716923] stm32-dwmac 5800a000.ethernet: Using 32/32 bits DMA host/device width
[ 4.790153] stm32-dwmac 5800a000.ethernet eth0: Register MEM_TYPE_PAGE_POOL RxQ-0
[ 4.798399] stm32-dwmac 5800a000.ethernet eth0: PHY [stmmac-0:00] driver [Generic PHY] (irq=POLL)
[ 4.811330] stm32-dwmac 5800a000.ethernet eth0: No Safety Features support found
[ 4.818694] stm32-dwmac 5800a000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[ 4.827408] stm32-dwmac 5800a000.ethernet eth0: registered PTP clock
[ 4.836131] stm32-dwmac 5800a000.ethernet eth0: configuring for phy/rgmii link mode
[ 4.847854] stm32-dwmac 5800a000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[ 4.854815] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

 

 Status; 

• Interface is up and responsive. It doesn't crash and detects a link automatically via Autonegotiate as expected.
• Data transfer DOESN't work (e.g. dhcp or ping with fixed IP) never any data is passed through. 

Anyone with hints / tips what to check? 

Dump of Information:

 

root@wesl:~# ifconfig
eth0 Link encap:Ethernet HWaddr 10:E7:7A:E1:85:E5 
 inet6 addr: fe80::12e7:7aff:fee1:85e5/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000 
 RX bytes:0 (0.0 B) TX bytes:3120 (3.0 KiB)
 Interrupt:50 

root@wesl:~# ethtool eth0
Settings for eth0:
	Supported ports: [ TP	 MII ]
	Supported link modes: 10baseT/Half 10baseT/Full
	 100baseT/Half 100baseT/Full
	 1000baseT/Full
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes: 10baseT/Half 10baseT/Full
	 100baseT/Half 100baseT/Full
	 1000baseT/Full
	Advertised pause frame use: Symmetric Receive-only
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Link partner advertised link modes: 10baseT/Half 10baseT/Full
	 100baseT/Half 100baseT/Full
	 1000baseT/Full
	Link partner advertised pause frame use: No
	Link partner advertised auto-negotiation: Yes
	Link partner advertised FEC modes: Not reported
	Speed: 1000Mb/s
	Duplex: Full
	Auto-negotiation: on
	master-slave cfg: preferred slave
	master-slave status: master
	Port: MII
	PHYAD: 0
	Transceiver: external
	Supports Wake-on: ug
	Wake-on: d
 Current message level: 0x0000003f (63)
 drv probe link timer ifdown ifup
	Link detected: yes
root@wesl:~# 

root@wesl:~# cat /sys/kernel/debug/clk/clk_summary | grep eth
 ethstp 0 0 0 266500000 0 0 50000 N
 ethmac 1 1 0 266500000 0 0 50000 Y
 ethtx 1 1 0 266500000 0 0 50000 Y
 ck_ker_eth 2 2 0 125000000 0 0 50000 Y
 ethptp_k 1 1 0 125000000 0 0 50000 Y
 ethck_k 2 2 0 125000000 0 0 50000 Y
 ethrx 1 1 0 125000000 0 0 50000 Y

 

mdio tool shows 2 phys (reporting on addr 4 AND addr 0)

 

root@wesl:~# mdio stmmac-0
 DEV PHY-ID LINK
0x00 0x00221652 up
0x04 0x00221652 up

oot@wesl:~# phytool eth0/0
ieee-phy: id:0x00221652

 ieee-phy: reg:BMCR(0x00) val:0x1040
 flags: -reset -loopback +aneg-enable -power-down -isolate -aneg-restart -collision-test
 speed: 1000-half

 ieee-phy: reg:BMSR(0x01) val:0x796d
 capabilities: -100-b4 +100-f +100-h +10-f +10-h -100-t2-f -100-t2-h
 flags: +ext-status +aneg-complete -remote-fault +aneg-capable +link -jabber +ext-register

 

[PatrickF]

Hi,

maybe a basic info:

For RGMII, did you enabled the ~2ns delay on RX and TX inside the PHY (sometimes it is thru HW config, sometimes thru MDIO SW control with device tree) ?

Regards.

[stefboerrigter]

Thanks @PatrickF .

We have not set any delay specifically. we just copied the implemenation from the DK boards. 
Also not sure if the path differences is wrong, according to the datasheet it is below <1ns differenence, therefore not a problem? but we can test if you give me some more info.

What we do notice, and are not sure about is the difference in TX and RX clock. We see in Linux that the RX and TX clock have different sources (and frequencies), and also when we measure them they are significantly different.
The RX clock is a nice 125MHz clock, but TX seems to be jittering all over the place (or we cannot measure it correctly). 

[PatrickF]

Hi,

delay on RGMII is a must, it could be either on GMAC or PHY side, but as STM32MP1 does not have embedded delay on neither RX or TX direction, it must be enabled on the PHY side for RX and TX (depend on the PHY driver I guess).

 

Regards.