STM32MP15 ECO 5.0.0 OPTEE in SYSRAM non-secure privileged write AXI ID 480

[debugging]

Same or related posts

https://community.st.com/t5/stm32-mpus-products/op-tee-error-on-new-v5-0-0-sdk/td-p/578239/page/2

https://community.st.com/t5/stm32-mpus-products/op-tee-error-tzc-permission-failure-on-new-v5-0-0-sdk/m-p/584339

Custom STM32MP157AAA3 board,  512MB. TF-A OP-TEE in SYSRAM

Two outcomes.

[a] building u-boot with stm32mp1_trusted_defconfig : boot loops around (restart), does not even reach u-boot

[b] building u-boot with stm32mp1_defconfig: dump_fail_filter:425 Violation @0xdfb19000, non-secure privileged write, AXI ID 480

This area  was firewalled in fw-config.dts based on the Wiki how to configure FW-CONF and also added as a reserved area in TF-A, OPTEE and u-boot DT,  Though u-boot (or something else for that matter) tries to access this area ?

Does this violation have anything to do with

a. conf.mk  in OP-TEE or the OP-TEE DT ?  (memory reservation was added)

b.  FW-CONFIG DT?  (Firewall for this region was added)

c.  U-BOOT  DT?  (the memory reservation was added)

d. Anything else ?

The U-boot WIKI states that stm32mp15_defconfig is to be used when when FIP is used in OpenSTLinux" and als that the two next defconfig are kept only for compatibility with upstream and should not be used:/ Those are basic & trusted

That leaves me with the memory violation issue.

No idea what else can be done to configure TZC and U-boot to solve this issues. Some help will be really appreciated Logs attached.

[debugging]

Because I have problem to get it run in SYSRAM (no help received so far) I changed my config to try to run in DDR and got the infamous error that the "entry  address not in reserved" area

My understanding, so far, is that there is a difference between running OP-TEE in SYSRAM or in DDR.

I found the TF-A STM32MP157-fw.config.dtsi  file always defaulted to OPTEE in SYSRAM  no matter OPTEE_IN_SYSRAM is "1" or not provided.  ST wrote that STM32MP15 default to DDR. This is true for OPTEE conf.mk (OPTEE_IN_SYSRAM?=n) but not for the TF-A fw-config device tree.

Finally I found the only way to get the TF-A device blob use the address from  DDR is on the make command line: STM32MP1_OPTEE_IN_SYSRAM=0.  Then the tos_fw entry changed to a DDR address instead of a SYSRAM entry address : 0x2ffc0000,

Furthermore , also with OP-TEE in DDR it seems that the calculations for the OPTEE entry address in  STM32MP157-fw.config.dtsi are incorrect. (see previous post), When using the original source the entry in the fw-config.dtb is  0xDC000000 while the build OP-TEE build  generates as entry address in the optee-header_v2.bin 0xDE0000000

fdtdump <<board>-fw-config.dtb> file:

tos_fw {
load-address = <0x00000002 0xde000000>;
max-size = <0x01e00000>;
id = <0x00000004>;
};

I think  that address should be the same as in the optee_header.bin file  (hexdump -C optee_header.bin)

That assumption was correct as the boot went trough with OPTEE in DDR. Though the AXI error 480 presists. the conclusion so far is that AXI Error 480 has not related by either using OPTEE in SYSRAM with paging in DDR or in completely in DDR. It's seems something else

This is when using OP-TEE completely in DDR, ID#9 BL32-extra is not loaded and BL32 extra-1 is fully loaded in DDR

INFO: Image id=4 loaded: 0xde000000 - 0xde00001c
INFO: OPTEE ep=0xde000000
INFO: OPTEE header info:
INFO: magic=0x4554504f
INFO: version=0x2
INFO: arch=0x0
INFO: flags=0x0
INFO: nb_images=0x1
NOTICE: DEBUG: parse optee image #0
NOTICE: DEBUG: header->optee_image_list[num].image_id=0x0
NOTICE: DEBUG: OPTEE_PAGER_IMAGE_ID=0
NOTICE: DEBUG: OPTEE_PAGED_IMAGE_ID=1
NOTICE: DEBUG: init_load_addr=0xde000000
NOTICE: DEBUG: init_size=0x3bb68
NOTICE: DEBUG: image_info->image_base=0xde000000
NOTICE: DEBUG: image_info->image_max_size=0x1e00000
INFO: BL2: Loading image id 8
INFO: Loading image id=8 at address 0xde000000
INFO: Image id=8 loaded: 0xde000000 - 0xde03bb68
INFO: BL2: Skip loading image id 9   >FIP  BL32 extra 2 (OPTEE pagable) is not loaded because all OPTEE is in DDR. (BL32 Extra 1) (OPTEE pager)
INFO: BL2: Loading image id 2
INFO: Loading image id=2 at address 0xc0500000
INFO: Image id=2 loaded: 0xc0500000 - 0xc0521630
INFO: BL2: Skip loading image id 16
INFO: BL2: Loading image id 5
INFO: Loading image id=5 at address 0xc0100000
INFO: Image id=5 loaded: 0xc0100000 - 0xc01f5d3c
NOTICE: BL2: Booting BL32
INFO: Entry point address = 0xde000000
INFO: SPSR = 0x1d3
I/TC: Early console on UART#4
I/TC:
I/TC: Embedded DTB found
I/TC: OP-TEE version: Unknown_3.19 (gcc version 12.2.0 (GCC)) #1 Sun Sep 24 00:47:59 UTC 2023 arm
I/TC: WARNING: This OP-TEE configuration might be insecure!
I/TC: WARNING: Please check https://optee.readthedocs.io/en/latest/architecture/porting_guidelines.html
I/TC: Primary CPU initializing
I/TC: WARNING: All debug access are allowed
I/TC: Platform stm32mp1: flavor PLATFORM_FLAVOR - DT stm32mp157a-board-mx.dts
I/TC: DTB enables console (non-secure)
I/TC: No power configuration found in DT
I/TC: Primary CPU switching to normal world boot
optee optee: OP-TEE: revision 3.19

U-Boot 2022.10-stm32mp-r1 (Sep 24 2023 - 08:48:35 +0800)

CPU: STM32MP157AAA Rev.B
Model: STMicroelectronics STM32MP157C eval daughter on eval mother
Board: stm32mp1 in trusted mode (st,stm32mp157c-ev1)
DRAM: 512 MiB
E/TC:0 tzc_it_handler:79 TZC permission failure
E/TC:0 dump_fail_filter:420 Permission violation on filter 0
E/TC:0 dump_fail_filter:425 DEBUG: status = 0x1
E/TC:0 dump_fail_filter:426 DEBUG: IT_STATUS = 0x10
E/TC:0 dump_fail_filter:427 DEBUG: filter = 0x0
E/TC:0 dump_fail_filter:428 DEBUG: tzc.base = 0xd8806000
E/TC:0 dump_fail_filter:429 DEBUG: FAIL_ID(filter) = 0x2c
E/TC:0 dump_fail_filter:431 Violation @0xdfb19000, non-secure privileged write, AXI ID 480
E/TC:0 Panic at core/arch/arm/plat-stm32mp1/plat_tzc400.c:84 <tzc_it_handler>
E/TC:0 TEE load address @ 0xde000000
E/TC:0 Call stack:
E/TC:0 0xde002645
E/TC:0 0xde01450b
E/TC:0 0xde00405d
E/TC:0 0xde014281
E/TC:0 0xde01f099
E/TC:0 0xde000348

 

 

[debugging]

Digging further, The AXI error happens in the u-bboot code, just after ./u-boot-stm32mp-v2022.10-stm32mp-r1-r0/build/stm32mp15_defconfig/source/common/board_f.c: function  announce_dram_init(void). The suspicion is that U-boot calls the OP-TEE service. (that is, it executes a trap instruction to that OP-TEE is called)

The AXI 480 crash happens after executing all steps in const init_fnc_t init_sequence_f[]  in
board_f.c after clear_bss().  jump_to_copy is never executed. The relocation of u-boot and dram initialization which is done in this code appears  to be fine.

#if !defined(CONFIG_ARM) && !defined(CONFIG_SANDBOX) && \
!CONFIG_IS_ENABLED(X86_64)
jump_to_copy,
#endif
NULL,

 

[debugging]

After  looking at someone else's good boot log, there is an optee-sign (see below) just after the DRAM size message (from the uboot source tree). This is EXACTLY the point where it, in my case, crashes with AXI 480 errror generated by the code in the  OP-TEE  source tree.  I can't find in the code where this happen other than exactly at the "jump"_to_copy in U-boot. Anyone an ideas ?

Added more DEBUG statement in the boot and enabled log messages. attached the log.

 I am wondering what const init_fnc_t init_sequence_f[] in uboot code is doing, If it's relocation then from where to where? a hypothesis is that its is relocating u-boot from the area where TF-A loaded it from the FIP

Loading image id=5 at address 0xc0100000

INFO: Image id=5 loaded: 0xc0100000 - 0xc01f5d3c.

and also uboot dtb from FIP

BL2: Loading image id 2
INFO: Loading image id=2 at address 0xc0500000
INFO: Image id=2 loaded: 0xc0500000 - 0xc0521630

Into the end of DDR  @@0xdfb****, which is an address in OPTEE 30MB secure region and then after the relocation (which seems successful) it jumps to the u-boot code in DDR.  Then TZC triggers an violation due to a write. It may be the TZC400 is incorrectly configured or the relocation address was incorrect.. If this hypothesis is correct, how to check the TZC settings ? There are set in FW-config firewall (see post before) and appear to be correct

I am quite sure someone from ST must have the knowledge and ideas or suggestion how to check this further.

 

U-Boot 2022.10-stm32mp-r1 (Oct 03 2022 - 19:25:32 +0000)

CPU: STM32MP151AAC Rev.Z
Model: STMicroelectronics custom STM32CubeMX board - openstlinux-6.1-yocto-mickledore-mp1-v23.06.21
Board: stm32mp1 in trusted mode (st,stm32mp151a-tios-mx)
DRAM: 512 MiB
optee optee: OP-TEE: revision 3.19 (afacf356)
Clocks:
- MPU : 650 MHz
- MCU : 200 MHz
- AXI : 266.500 MHz

Looking at the log:

 

Ram top: E0000000 => correct
initcall: c0120669
initcall: c0101d0f
initcall: c01209ed
Reserving 4032k for video at: dfc00000  => optee framebuffer ?? If I understood this should be 0xDD0000000
initcall: c0120af9
initcall: c0120889
Reserving 953k for U-Boot at: dfb11000  =>in the 30MB OPTEE area form 0xDE000000~DFE0000000 ??
initcall: c0120bad
Reserving 32776k for malloc() at: ddb0f000  => u-boot DTB ??
initcall: c0120b01
Reserving 68 Bytes for Board Info at: ddb0efb0  => Frame buffer area @ DD0000000 ??
initcall: c0120bf9
Reserving 296 Bytes for Global Data at: ddb0ee80
initcall: c0120819
Reserving 136768 Bytes for FDT at: ddaed840
initcall: c0120b5d
Reserving 0x278 Bytes for bootstage at: ddaed5c0

same here:

New Stack Pointer is: ddaed5a0
New Stack Pointer is: ddaed5a0
initcall: c01206c9
init_func_watchdog_reset
initcall: c0120719
DEBUG: reloc_fdt - startedn
DEBUG: reloc_fdt - ended
initcall: c0120935
DEBUG - reloc_bootstage - started
Copying bootstage from c0080018 to ddaed5c0, size 278    => what is this "Bootstage ? "
DEBUG - reloc_bootstage - ended
initcall: c0120691
DEBUG - reloc_bloblist - started   => is bloblist DTB of u-boot ?
DEBUG - reloc_bloblist - ended
initcall: c012075d
DEBUG - setup_reloc - started
Relocation Offset is: 1fa11000   => v.s 0xC00000000
Relocating to dfb11000, new gd at ddb0ee80, sp at ddaed5a0   => what is gd ? what is relocated here ? this should be u-boot . but then what is "bootstage"
DEBUG - setup_reloc - ended

It seems u-boot using different address that defined in the TF-A/u-boot/OPTEE device trees and TF-A fw-config ? It then relocated itself into the 30MB OP-TEE area (0xDE000000 size 0x1E00000) and that may be the reason for the TZC error.

And why does the the copy (write) into 0xDFBxxxxxx for boot stage ( u-boot code?) goes ok, if it is OPT-EE TZC protected  and where does u-boot get these address from ? 

[debugging]

Looking for clues for the uboot address, In the u-boot stm32mp15_defconfig found this line .

CONFIG_DEFAULT_DEVICE_TREE="stm32mp157c-ev1"

Assumed this might be incorrect. When changing  this to the CubeMX generated device tree file name, the boot keep cycling in a reboot and the relocation does not even start Thus things got worse.   there is nothing else related to memory in the defconfig other then the default 0xC0000000

 

 

[debugging]

Created a whole new ECO 5.0 file tree (unzip from all sources) and started from scratch using the stm32mp157a-dk1 device trees.  Updated the OPTEE device tree  and removed all unused devices, such as SAI, LCD, unused UARTs, SPI, I2C, I2S including I2C4 for PMIC and power management entries to obtain a minimal configuration with just  UART4 and SDMMC1. Ultimately, bumped into exact same AXI 480 error.  Line 82 in OP-TEE tzc400.c reported the panic, but how to check what generated it. ?

if (IS_ENABLED(CFG_STM32MP_PANIC_ON_TZC_PERM_VIOLATION)) {

stm32mp_dump_core_registers(true);

panic();

} else {

[debugging]

After one month of effort  to try to bring up the board with ECO 5.0.0 this is the last post, I have to give up. w/o any clues or direction (other than... read the WIKI), it's no use. it's a pity, spend months on the hardware, but first need to be able to build an boot chain on a known good reference board.

#end of post

[debugging]

Started with a new set of ECO 5.0.0 and Cube-MX files. Did not apply the 2MB reserved area in the fw-config firewall this time, The AXI 480 error appeared again. After adding the optee reserved area in U-boot.dts the AXI  error went away.  This thus AXI 480 is caused by missing optee-reserved area in u-boot DTS. While this error is generated by OP-TEE code, it happens at u-boot execution. My hypothesis is that of u-boot is using OP-TEE services.