STM32MP15 ECO 5.0.0 BL2 stuck at loading BL32

[debugging]

Custom STM32MP157AAA3 board with 512MB DDR.

[Note: the NOTICE: DEBUG messages in below logs were added to the source code of TF-A]

The TF-A DT was compiled with option for SYSRAM with resulting dtb fw-config

OP-TEE was compiled for SYSRAM and and enabled with pager as can be seen in  the contents FIP file:

The reservation was made in OP-TEE DT:

The optee-header_v2.bin: file  generated:

Result:

NOTICE: Early console setup
NOTICE: CPU: STM32MP157AAA Rev.B
NOTICE: Model: STMicroelectronics custom STM32CubeMX board - openstlinux-6.1-yocto-mickledore-mp1-v23.06.21
NOTICE: DEBUG: stm32_print_boardinfo
NOTICE: DEBUG: stm32mp_is_closed_device
NOTICE: HELLO: fixed_regulator_register
NOTICE: DEBUG: stm32mp1_arch_security_setup
NOTICE: DEBUG: print_reset_reason
INFO: Reset reason (0x14):
INFO: Pad Reset from NRST
NOTICE: DEBUG: stm32mp1_syscfg_enable_io_compensation_finish
NOTICE: DEBUG: fconf_populate
INFO: FCONF: Reading TB_FW firmware configuration file from: 0x2ffe2000
INFO: FCONF: Reading firmware configuration information for: stm32mp_io
NOTICE: DEBUG: stm32_skip_boot_device_after_standby
NOTICE: DEBUG: stm32mp_io_setup
INFO: Using SDMMC
INFO: Instance 1
INFO: Boot used partition fsbl1
NOTICE: BL2: v2.8-stm32mp1-r1.0(debug):()
NOTICE: BL2: Built : 15:24:01, Sep 11 2023
INFO: BL2: Doing platform setup
INFO: RAM: DDR3-DDR3L 16bits 533000kHz
INFO: Memory size = 0x20000000 (512 MB)
INFO: BL2: Loading image id 1
INFO: Loading image id=1 at address 0x2ffff000
INFO: Image id=1 loaded: 0x2ffff000 - 0x2ffff1ea
INFO: FCONF: Reading FW_CONFIG firmware configuration file from: 0x2ffff000
INFO: FCONF: Reading firmware configuration information for: dyn_cfg
INFO: FCONF: Reading firmware configuration information for: stm32mp1_firewall
INFO: BL2: Loading image id 4
INFO: Loading image id=4 at address 0x2ffc0000
INFO: Image id=4 loaded: 0x2ffc0000 - 0x2ffc002c
INFO: OPTEE ep=0x2ffc0000
INFO: OPTEE header info:
INFO: magic=0x4554504f
INFO: version=0x2
INFO: arch=0x0
INFO: flags=0x0
INFO: nb_images=0x2
NOTICE: DEBUG: parse optee image #0
NOTICE: DEBUG: header->optee_image_list[num].image_id=0x0
NOTICE: DEBUG: OPTEE_PAGER_IMAGE_ID=0
NOTICE: DEBUG: OPTEE_PAGED_IMAGE_ID=1
NOTICE: DEBUG: init_load_addr=0x2ffc0000
NOTICE: DEBUG: init_size=0x26fc0
NOTICE: DEBUG: image_info->image_base=0x2ffc0000
NOTICE: DEBUG: image_info->image_max_size=0x22000
WARNING: The load address in optee header 0x2ffc0000 - 0x2ffe6fc0 is not in reserved area: 0x2ffc0000 - 0x2ffe2000.
ERROR: OPTEE header parse error.
PANIC at PC : 0x2ffed03d

The OP-TEE generated header file optee-header_v2.bin has the load address for FW_CONFIG (image ID 1) set to 0x2FFC0000,  that is the same address that OPTEE tries to load loading Image ID  #8 (extra 1) , also the boot log below. Changed the starting address in the OPTEE , conf.mk file to 0x2FFC0200 to avoid a conflict. (shown in the picture below).  The optee-header_v2.bin has the size set to 0x26dc0 which is the size of BL32 Extra 1  (ID 8) this isis too large to load in SRAM and created an BL2 load error as above.

After changing the size in the OP-TEE .mk file to to 0x1000 the OPTEE build still set it to 0x26EC0 in the header.:

It seems the OPTEE build  ignored the config.mk file setting

OP-TEE conf.mk file:

CFG_TZSRAM_START ?= 0x2ffc0200
CFG_TZSRAM_SIZE ?= 0x00010000

The next attempt to load ID 8 (BL32 extra 1) was to manually changing the header file to size of 0x1000. 

The changed header as follows

This went through a bit more but an “out of bounds" error is raised. See the log below.

Questions:

1 Why does OPTEE use the same start address for image ID 8 (Extra 1) and for ID 1 (FWCONFIG) ? Is this correct ? If we had not changed the base address in the conf.mk to 0x2ffc000200 it would start to overwrite FWCONFIG data at 0x2ffc0000 with ID8  BL32,  OP-TEE extra 1, data

2. Why does OPTEE set a larger size in the header for BL32 extra 1 than available in SRAM for OT-TEE ?

3. and why does OPTEE ignore the CFG_TSZR_RAM size setting t(to reduce the size for SYSRAM) in the conf.mk file ?

4. Where does BL2 get the base address 0xde000000 and size from (see log below) ?

5. Is 0xde000000 the address BL2 will try to load image ID#9 (BL32 Extra 2) ?

6. What is/could be wrong ?

7. is there a "good boot" log of an DK or EV board using ECO 5.0.0 to see what the  memory locations are  used by BL2 for loading the FIP images ?

Boot log after manually changing the size to 0x10000 in the header:

INFO: Using SDMMC
INFO: Instance 1
INFO: Boot used partition fsbl1
NOTICE: BL2: v2.8-stm32mp1-r1.0(debug):()
NOTICE: BL2: Built : 15:24:01, Sep 11 2023
INFO: BL2: Doing platform setup
INFO: RAM: DDR3-DDR3L 16bits 533000kHz
INFO: Memory size = 0x20000000 (512 MB)
INFO: BL2: Loading image id 1
INFO: Loading image id=1 at address 0x2ffff000
INFO: Image id=1 loaded: 0x2ffff000 - 0x2ffff1ea
INFO: FCONF: Reading FW_CONFIG firmware configuration file from: 0x2ffff000
INFO: FCONF: Reading firmware configuration information for: dyn_cfg
INFO: FCONF: Reading firmware configuration information for: stm32mp1_firewall
INFO: BL2: Loading image id 4
INFO: Loading image id=4 at address 0x2ffc0000
INFO: Image id=4 loaded: 0x2ffc0000 - 0x2ffc002c
INFO: OPTEE ep=0x2ffc0000
INFO: OPTEE header info:
INFO: magic=0x4554504f
INFO: version=0x2
INFO: arch=0x0
INFO: flags=0x0
INFO: nb_images=0x2
NOTICE: DEBUG: parse optee image #0
NOTICE: DEBUG: header->optee_image_list[num].image_id=0x0
NOTICE: DEBUG: OPTEE_PAGER_IMAGE_ID=0
NOTICE: DEBUG: OPTEE_PAGED_IMAGE_ID=1
NOTICE: DEBUG: init_load_addr=0x2ffc0200
NOTICE: DEBUG: init_size=0x10000
NOTICE: DEBUG: image_info->image_base=0x2ffc0000
NOTICE: DEBUG: image_info->image_max_size=0x22000
NOTICE: DEBUG: parse optee image #1
NOTICE: DEBUG: header->optee_image_list[num].image_id=0x1
NOTICE: DEBUG: OPTEE_PAGER_IMAGE_ID=0
NOTICE: DEBUG: OPTEE_PAGED_IMAGE_ID=1
NOTICE: DEBUG: parse_optee_image

NOTICE: DEBUG: init_load_addr=0xde000000
NOTICE: DEBUG: init_size=0x84000
NOTICE: DEBUG: image_info->image_base=0xde000000
NOTICE: DEBUG: image_info->image_max_size=0x2000000
INFO: BL2: Loading image id 8
INFO: Loading image id=8 at address 0x2ffc0200
WARNING: Image id=8 size out of bounds
ERROR: BL2: Failed to load image id 8 (-27)

 

[debugging]

The reason is was the DEBUG=1, and all other DEBUG options in TF-A/TF-FWCONFIG such as ST_TF_A_LOG_LEVEL_RELEASE=LOG_LEVEL_INFO and TEE_CORE_LOG_LEVEL=3 ST_OPTEE_DEBUG_LOG_LEVEL=3   in OPTEE. When removed it used the correct load addres image_base=0xde000000 instead of image_info->image_base=0x2ffc0000. (Note This meant also  TF-A creates the output in RELEASE folder, not debug)

 

[damien1]

There is quite a bit of conflicting info in the wiki when it comes to using the latest ECO release.

From what I've read it is no longer required to store Optee in SRAM in the latest release and this is good because Optee never really fit into the SRAM anyway and used the pager to put some of itself in DRAM.

CFG_DRAM_SIZE ?= 0x10000000
# ifeq ($(CFG_STM32MP15)-$(CFG_WITH_PAGER),y-y)
# CFG_TZSRAM_START ?= 0x2ffc0000
# CFG_TZSRAM_SIZE ?= 0x0003f000
# endif #CFG_STM32MP15 && CFG_WITH_PAGER
CFG_TZDRAM_SIZE ?= 0x02000000
CFG_TZDRAM_START ?= ($(CFG_DRAM_BASE) + $(CFG_DRAM_SIZE) - $(CFG_TZDRAM_SIZE))

 

[debugging]

<updated>

Nice to see more folks follow this :) From ECO  5.0. STM decided to  default from SRAM back to DRAM. On STM32MP15 It can use paging swapping with main portion loaded in DRAM data . Running in DRAM is un-secure according to the WIKI so I would guess that is not recommended for actual commercialized products .

[Bernard PUEL]

You are perfectly right Damien. With V5 came a major change came in: by default OP-TEE will run in DDR.

No issue for STM32MP13 because a HW encryption allow to encrypt OP-TEE in ddr to run secure services. But for STM32MP15, if you need secure services, you will need to go back to internal RAM: This is explained here: https://wiki.st.com/stm32mpu/wiki/How_to_configure_OP-TEE#OP-TEE_core_configuration

Depending on your product, you may use more internal RAM than SYSRAM to avoid to use optee pager that will introduce some latency in your system (but may be not an issue for your product).

For this the constraint is to map OP-TEE internal RAM in a contiguous area. See this OP-TEE community thread on this subject if it is your interest: https://github.com/OP-TEE/optee_os/issues/5971 

[debugging]

@Bernard PUEL  Thank you, but I get very few feedback on community posts to ask for example "did you did that, or this" to nail down the cause. I could not get the board to boot w/o compiling TF-A & OPTEE in SYSRAM and only managed to get the board to pass OPTEE loading after finding the mistake in the WIKI (lost a lot of hairs and sleeping hours).

What are the (commercial) product implications/limitations when running OPTEE in DDR w/o SRAM ? Is there are pro/con WIKI ?

[Bernard PUEL]

You are welcome. IT should be easier to keep OP-TEE running in DDR. I understand you have a 512MB Memory size, so if you copy/past ST DK Board memory mapping, it should be obvious. Note the "should" ...
By the way Security policy is really customer choice for his product but ST position (that is obvious position) is to say that running secure services is an non secure external memory makes no sense.

But OP-TEE is not running only secure service but also system services (like PSCI or SCMI services, ...).

That's what reflects this table I mentioned above: https://wiki.st.com/stm32mpu/wiki/How_to_configure_OP-TEE#OP-TEE_core_configuration

So for example if you use Trusted applications for your product, It is because you want this service to be secure and therefore, you should not run it in non encrypted DDR.

[Bernard PUEL]

Product security is a big and complex subject really depending on the product. This needs to be global (HW + SW) and coherent from the production of the product and also after life of the product.

 

You can have an overview of the different tools available on STM32MP1 = https://wiki.st.com/stm32mpu/wiki/Security_overview

 

You may also request help from partners (like proven run).

 

After you decided which OP-TEE services are mandatory for your product, table https://wiki.st.com/stm32mpu/wiki/How_to_configure_OP-TEE#OP-TEE_core_configuration is here to advice between the 2 choices (internal or external RAM).

[debugging]

SCMI and PSCI are also supposed to run protected, to protect the system.  Thus running in DDR is not good choice for this. correct ? And if so, a commercial product should run OP-TEE in SRAM. Why did ST choose to run in in unsecure DDR as default ? + It would be helpful to have, in layman's terms, a description about  the risks of running OP-tee in un-secure DDR for product designers to make a calculated decision. For example one could use an external secure element to store secrets.

 

 

[Bernard PUEL]

You are right, non secure services were moved to OP-TEE to protect the system. This is not mandatory for any product, again this is customer choice. 

Then secure services may access to secret that could have more impact (for example for payment terminal) on other systems than just the product itself. Usually these systems (secure ones) have different levels of security and in that case, they prefer to kill the product if they detect security attacks (like TAMPER detection) physically on the product.

 

Both OP-TEE in DDR or Internal RAM is a valid use case for a commercial product.

[debugging]

Thank for the reply, but still wondering what are the key selection criteria to make a decision between OP-TEE in DDR or internal RAM ?

[debugging]

i assume this should be : this is not mandatory for every product  

It depends the use case, . In other words, if the product does need the protection of malicious attacks just use DRAM.

 

[Bernard PUEL]

Here is also a high level introduction if needed = https://www.st.com/content/st_com/en/premium-content/premium-content-white-paper-protecting-oem-assets-and-ensuring-firmware-integrity-through-stm32-and-open-source-solutions.html

[debugging]

Thank for the feedback. Still stuck at the AXI 480 error. See my post. This happens  both  with OPTEE in DDR or in SRAM, I can switch between those quickly using custom build boot scripts. Both  results in same problem and it has taken me a month now to boot a simple board with discrete power and still unsuccessful. Spending many nights and days on this. I have no clues anymore. :slightly_frowning_face: