Is NASM inconsistent or am I just missing an obvious fact with CMP of immediate?

Question

The "warning: signed dword immediate exceeds bounds" is the bane of my existence at the moment as it appears to be inconsistent or I am just not seeing an obvious fact.

I have the following structure declared:

struc   FRTType        
        .class    resq  1   ; Class
        .type     resq  1   ; Type
endstruc 

I have the following assigns:

%assign TYPE_SCALAR     0xfffffffffffffff1
%assign INTEGER         0xffffffff1000a8a9

And in a function I have:

cmp     qword [rdi+FRTType.class], TYPE_SCALAR  ; This works fine
jne     .exception
cmp     qword [rdi+FRTType.type], INTEGER       ; THIS PRODUCES WARNING

I know I can mov rax, INTEGER and then do the compare but that seems unneeded given the first compare has no problem.

Answer 1

There's no CMP r/m64,imm64 .
There's CMP r/m64,imm32 , where imm32 is sign-extended to 64 bits. Which works fine for 0xfffffffffffffff1 , because 0xfffffff1 sign-extended to 64 bits is 0xfffffffffffffff1 . But 0x1000a8a9 sign-extended to 64 bits is 0x000000001000a8a9 , which differs from the value you wanted to compare against.

You could overcome this eg by loading the immediate into a register first:

mov rax, INTEGER
cmp     qword [rdi+FRTType.type], rax