Reverse engineer array dimensions / struct layout from compiler asm output?

Question

In this code, A and B are constants defined with #define . What are the values of A and B?

typedef struct {
    int x[A][B];
    long y;
} str1;

typedef struct {
    char array[B];
    int t;
    short S[A];
    long u;
} str2;

void setVal(str1 *p, str2 *q) {
    long v1 = q->t;
    long v2 = q->u;
    p->y = v1+v2;
}

The following assembly code is generated for the setVal procedure:

setVal:
    movslq  8(%rsi), %rax
    addq   32(%rsi), %rax
    movq     %rax, 184(%rdi)
    ret

Answer 1

The structure has the following alignment requirements:

• a char may start at any byte
• a short may start at even byte
• an int may start at byte, divisible by four
• a long may start at byte, divisible by eight

The str1.y field is a long and starts at 184 , this implies, that str1.x may hold either 184 or 180 bytes.

The str2.t field is an int and starts at 8 , this implies, that str1.array may hold from 5 to 8 bytes.

The str2.u field is a long and starts at 32 , this implies, that str2.S may hold from 14 to 20 bytes.

This is the diagram for str1 structure fields:

+---------------+---+--------+
|  int x[A][B]  | ? | long y |
+---------------+---+--------+
|        184        |    8   |
+-------------------+--------+

And this is the diagram for str2 fields:

+---------------+---+-------+------------+---+--------+
| char array[B] | ? | int t | short S[A] | ? | long u |
+---------------+---+-------+------------+---+--------+
|         8         |   4   |        20      |    8   |
+-------------------+-------+----------------+--------+

After that, you should solve the following system:

177 <= 4 * A * B <= 184
5 <= B <= 8
14 <= A * 2 <= 20      // 7 <= A <= 10

The answer is: A = 9 , B = 5

---

You can test your answer (and the ranges for each inequality) using a compiler that follows the same ABI / calling convention used by the compiler that produced the original code. It uses 8-byte long : note the 64-bit operand size for the addq , instead of addl , and the 8-byte store. Thus, we can infer that it's most likely the x86-64 System V ABI, not the Windows x86-64 calling convention (which uses 4-byte long ).

The Godbolt compiler explorer has gcc, clang, ICC, and MSVC. The first 3 target Linux, but MSVC targets the Windows calling convention and thus won't agree on struct layout with a smaller long requiring less alignment.

Replacing int x[A][B] with char t[177] (or other sizes) proves that 177 is the minimum and 184 is the maximum size that that leads to a store to 184(%rdi) . So we could have written 176 < 4*A*B <= 184 . Or, to keep things to multiples of 4, 180 <= 4*A*B <= 184 is also more or less correct; we can rule out 177..179 based on the size of int .