为什么 JL 命令在我的汇编代码中不起作用？

Question

In my assembly program I am interested to know what's the minimal value %rsp gets (as it grows down) compared to its initial value so I wrote in main:

mov %rsp, %rcx #start
mov %rsp, %rdx #max

and in every part in my code before push (or any other command that may affect %rsp) I wrote:

cmp %rsp, %rdx
jl next3 # current rsp is less that prev_max then skip the update of value for max
mov %rsp, %rdx
next3: # every time this is copied I change the number like next4, next5 etc...

but when I debug my code rdx and rcx share the same value, why is that?

Answer 1

Your stack checking code is in fact correct. Since you provided full code[*] we can see there is a problem starting from line 73:

    cmp (%rdi), %esi
    
    cmp %rsp, %rdx
    jl next4
    mov %rsp, %rdx
    next4:
    
    jne continue

Your stack check is in between the cmp (%rdi), %esi and the jne continue so it will use the flags from cmp %rsp, %rdx instead. As it happens those are equal at that point so your function will not go to continue but return. The stack space used will just be 16 bytes. Note this stack check block is useless as the stack pointer can not change from the previous one so you can remove lines 74-80. This will then produce a stack usage of 0x40 bytes as you expect.

As to the question of operand order, at&t uses the reverse to intel. Consider this code:

mov $1, %eax
mov $2, %edx
cmp %eax, %edx
jl next

This will not jump.

---

[*] https://onlinegdb.com/5CKsK1GNT with ja changed back to jl