Skip to main content
Graduate
January 23, 2023
Question

C / C++ memory layout and optimization, problem with type punning, loading and saving.

  • January 23, 2023
  • 5 replies
  • 5123 views

I have some data as static class members. I use `__attribute__((__packed__))` for some classes and structs. I use type punning to convert between types that I believe have the same memory layout, but different signatures (members of different translation units). It works. I assume the sequence and offsets of members is as in my class / struct definition. Knowing this I just save and load memory regions to and from files and it works.

Until I enable any optimization in the compiler. I noticed that when my configuration file was written by the debug version, it does not load in release version. The first member is loaded correctly, the subsequent ones are not.

My only guess here it can be related to a different memory layout using -O1 .. -O3. Can `__attribute__((__packed__))` be ignored when optimizations enabled? Is there a way to express in my code I want a specific memory layout for a structure?

When I have a class that has only static members, is it equivalent of a C struct in memory? When the class is marked with the attribute, does it affect its static members? Is it a way to ensure specific members are always put in specific relative memory locations?

Consider the code:

#include "stdint.h"
 
class __attribute__((__packed__)) X
{
public:
 static uint8_t a;
 static uint16_t b;
 static uint32_t c;
};
 
uint8_t X:a;
uint16_t X:b;
uint32_t X:c;
 
struct __attribute__((__packed__)) Y
{
 uint8_t a;
 uint16_t b;
 uint32_t c;
};
 
int main(void)
{
 Y s;
 s.a = 1;
 s.b = 2;
 s.c = 3;
 memcpy((void*)X.a, &s, sizeof(Y));
 bool ok = s.a == X.a && s.b == X.b && s.c == X.c;
}

When compiled with -O0, the `ok` is true. When compiled with -O1 - the `ok` is false.

What am I missing here?

    This topic has been closed for replies.

    5 replies

    Graduate II
    January 23, 2023

    >>I noticed that when my configuration file was written by the debug version, it does not load in release version.

    I fight with this every day.

    there is a mess between configuration for debug build/release builds, it gets worse if you changed the name of the project at any point.

    maybe this helps you

    https://community.st.com/s/question/0D53W000011upRKSAY/how-can-i-get-packed-attribute-to-work-in-the-gcc-compiler-used-in-the-stm32cubeide

    HTDAuthor
    Graduate
    January 23, 2023

    Wow, how did you guess that I've changed my project's name? ;) However, no file location issues so far, however, the __packed__ attribute seems to work weird. Thanks for the hint, I'll just code a test module and add it to my target project to be able to quickly test various configurations. I will test type punning and `memcpy()` - if the data are where I expect them to be.

    BTW, changing the project's name works only with editing the .project (XML) file, doing that in STM32CubeIDE or worse, in TouchGFX initiates the project's self-destruct sequence ;) So, just change project tag content, reload everything, then replace all the references and some configuration (build) settings manually. Works as charm. However, with the compiler optimization there's something odd going on and I will figure it out today.

    HTDAuthor
    Graduate
    January 23, 2023

    Unfortunately, that's not the problem. Packed attribute (and `#pragma pack(push, 1) / #pragma pack(pop)`) works regardless of optimization options. The struct data are packed, however, the ORDER of variables is changed!

    Here are my static members in order of definition in class header:

    0693W00000Y8wZVQAZ.pngNotice the weird address assignments! Without optimization the order of the static members is as defined in the header.

    The obvious fix for it is to create a separate structure for the data and use that structure for dumping data to a file, but that's adding probably unnecessary complexity to a simple piece of code.

    Super User
    January 23, 2023

    IMHO your code in lines 3-13 is er... to say it mildly ....

    A POD type with all members public in C++ is called 'struct', no members need to be static.

    But the usage of __attribute__((packed)) in gcc has non-intuitive subtleties. Modern gcc versions issue warnings when the __attribute__((packed)) is misplaced (and not honored). Do not ignore these warnings.

    Maybe pragma pack is easier to use, as in the thread quoted by @Javier Muñoz​ 

    Layout of plain old data types (structs/unions) does not depend on optimization level alone. Compiler folks do not dare to do that, otherwise angry users would come to them with fire and pitchforks.

    Type punning can be broken by optimization (if you know what it is, you know why it is questionable).

    HTDAuthor
    Graduate
    January 24, 2023

    It was the problem old as dinosaurs ;) I had a piece of code working as a tool (thus static) and it had some stored data (configuration). In a hurry, I mixed it in a way they never should be mixed. The solution was just to use a packed struct with the data. I also made a little test to see what happens with the memory layout.

    > Compiler folks do not dare to do that, otherwise angry users would come to them with fire and pitchforks.

    Exactly! So packed (1) is packed (1), always. Thanks for reminding about compiler warnings when packed attribute is misplaced. Pragma pack doesn't seem to emit warnings. However, unlike __attribute__((packed)), it's valid in MSVC, so I can use it in code that is compiled in VS.

    Also, memcpy() emits a warning. For this reason exactly.

    Super User
    January 23, 2023

    GCC can/will change the order of variables in memory depending on optimization levels. I hit this issue with the AVR GCC when allocating variables in the EEPROM space. Data written when compiled with "-Og" were not readable with code compiled with code compiled with "-Os". The solution, as you mention above, was to declare a structure that filled the entire EEPROM space and put the variables in the structure. The order inside the structure is guaranteed by the C/C++ spec.

    I haven't seen this issue with the STM32, but none of my STM32 code attempts to do what you are doing. I suspect you are seeing that same issue, with the three static members of X being placed in memory in a different order with different optimizations.

    Super User
    January 24, 2023

    I don't get the semantics of the original code, did it ever compile? Lines 11..13 in particular? What's the purpose?

    hth

    KnarfB

    HTDAuthor
    Graduate
    January 24, 2023

    Line 13 before fixing - not ;) It's a bad example. The problem with it, beside it illustrates a bad idea - is (after fixed `uint16_t` to `uint32_t`) it compiles, runs and `ok` is `true`. It happens because the compiler has no reason to reorder the static members in memory. In my real world code reordering occurs. Static members are fine, but they are something completely different from regular struct members. When no optimization is used, they behave exactly as the class was a singleton instance. With optimization the difference becomes apparent.

    Super User
    January 24, 2023

    > Static members are fine, but they are something completely different from regular struct members

    Yes, that's true. For a global variable (static or not), the final address is assigned by the linker, not the compiler.

    What is uint8_t X:a; ? Syntax error? Did you mean X::a; ?

    Don't see a use of defining three individual variables instead of one instance of X.

    hth

    KnarfB

    Visitor II
    December 29, 2024

    The issue is at least that a, b and c are static members of X. Static members do not affect the memory layout of an instance of a class.

    In your example, you try to memcpy sizeof(Y) bytes starting at the address of a into X::a. X::a is static and the only requirement concerning the memory layout for a static member stated in the standard is that it is located at a globally accessible address. This means that the memory following X::a can be whatever you can think of. It may be X::b, it may be X::c, it may be trash, it may be the address of something else in your program or even a memory address that you are not allowed to access (-> segmentation fault). What I mean by that is that you memcpy s into X::a followed by arbitrary memory. You get completely undefined behaviour at runtime.

    Since you are copying data anyway, serialize and deserialize functions may be worth to guarantee data consistency.