how to detect which line of code has modified the global variable (not debug mode)

[perlman]

Development environment: STM32F417ZG/keil uvision 5/FreeRTOS

Problem description: During the normal working phase of my product, a global variable is suddenly found to have been modified accidentally. This variable was not modified by my normal code as there is only one place where the normal code is modified and I am sure enough it doesn't get triggered. So it could have been caused by a memory stepping, thread safety or some other exception.

This is a very low probability problem, so it is difficult to reproduce the problem in keil's debug mode and use data breakpoint to locate the problematic code that modified this global variable.

My question is: how can I locate which line in my project's source code caused the global variable to be modified during the normal working phase of the product.

Does anyone have any good ideas on how to do this?

[mƎALLEm]

It could be caused by an overlapping write. For example you keep writing to a table and you exceed its border where your variable is located in that memory region. 
Or it could be a DMA access that overwrites your variable.

[perlman]

My goal is to locate the code line number that modifies the global variable. So it's crucial to notify me as long as the variable is modified.

I am not quite sure whether the following way is good:

As I am using STM32F417ZG that has MPU, it's possible to configure MPU to monitor the modified memory.

Once it's modified, a function like Memory_Fault_Handler() would be called. Inside the function some useful details including task-id can be set.

[mƎALLEm]

Yes I know, just wanted to give some tips on which scenario could occur. The MPU could be a possibility but its granuratity could be a problem as the min size is 32 bytes. So memory fault could be raised even your variable was not overwritten.

[Danish1]

Setting a breakpoint-on-write is the only way to know for certain what processor-instruction wrote to a specific memory-location. But don’t forget that DMA can also write to memory; how strongly do you know that isn’t the problem?

Facing a similar problem, I might see if anything can be learned from the new/corrupted value. Is this value out-of-range? Is it e.g. an ascii character? I would need a way to capture the event; maybe store things in non-volatile memory or written to a log file so they may be reviewed and analysed later. But simply displaying a message on screen or coded LED pattern for the user to note and acknowledge before your recovery code takes over might be an option.

”Unexpected” writes could be from buffer or stack overflow. I like to put “guard words” immediately beyond both ends of each stack and pre-fill the stack with a known value so I can measure how much stack gets used and quickly check for overflow (not definitive). But I would also put a few guard words either side of this key variable, preloaded with a known value and look to see if they also get overwritten.

it could be a miscalculated length in a memcpy; trying to put a string into a circular buffer and failing to handle the wrap.

Just a few ideas…

[AScha.3]

If you have SWO connected, you could try with SWD to catch the writing to the variable address of interest :

https://www.codeinsideout.com/blog/stm32/swv/#inspect-variables

 

[Pavel A.]

@perlman This is very easy. The ARM toolchain has the addr2line tool (arm-none-eabi-addr2line), it finds the given code address in the debug info of your program and gives you the file and line number. All that is left is to find the address of instruction that modifies your variable. Good luck.