'sprintf' libc call gives SegFault in assembly on Ubuntu (sometimes)

Question

I am trying to stringify and integer in assembly (NASM) by making an sprintf call.

I have the following assembly code which does this:

sprintf_Int_str db "%lld",
                   dq 0
...

Int.str:
    push rbp
    mov rbp, rsp  ; creates stack frame

    ; allocate string
    mov rdi, 64   ; much larger than needed
    call malloc

    push rax ; save char*


    ; write string to allocated memory using `sprintf(buf, "%lld", n)`
    mov rdi, rax
    mov rsi, sprintf_Int_str
    mov rdx, qword [rbp + 16]     ; the location of the number on the stack 
                                  ; (trust me - it's pushed before Int.str is called,
                                  ; meaning it's 2 (64-bit) places before the start 
                                  ; of the stack frame)
    mov rcx, rsi

    mov r8, 0
    mov r9, 0    ; clearing other registers for sake of it
    mov rax, 0   ; (doesn't make a difference if I do this or not)

    call sprintf

    pop rax      ; pop return value which is saved after call to malloc


    mov rsp, rbp
    pop rbp
    ret

This code works fine on my machine (Ubuntu 22.10), but I noticed that my automated tests were failing on GitHub Actions (Ubuntu VM). I set up some tests in Docker to run on my machine, and they worked fine in Alpine, Debian and Arch containers, but again failed in Ubuntu. I used GDB to analyse the core dump from the failing code, and it gave me this:

Which is not very helpful. I've tried all sorts of ways of passing the parameters to sprintf as I have seen conflicting info online, but GCC give the following output from the following C++ code:

char* Int_str(int i)
{
    char* str = malloc(64);
    sprintf(str, "%d", i);
    return str;
}

.LC0:
   .string    "%d"
   .text
   .globl Int_str
   .type  Int_str, @function
Int_str:
   pushq  %rbp
   movq   %rsp, %rbp

   subq   $32, %rsp
   movl   %edi, -20(%rbp)
   movl   $64, %edi
   call   malloc@PLT

   movq   %rax, %rdi
   movl   -20(%rbp), %edx
   leaq   .LC0(%rip), %rcx
   movq   %rcx, %rsi
   movl   $0, %eax
   call   sprintf@PLT

   movq   -8(%rbp), %rax

   leave
   ret

which works fine, and seems to be the same as my code, in terms of what registers have what in when sprintf is called.

What is particularly puzzling to me is how rarely it fails: always on a Ubuntu VM, never on my Ubuntu machine. My only idea is that I'm doing something with undefined behavior which happens to always work on my machine due to more RAM or more resilient something , but, on the more lightweight VM, that isn't allowed (although adding --max-ram="1g" doesn't help).

My assembling command is

nasm -f elf64 <file.asm> -o out.o

And my linking command is

gcc -Wall -no-pie out.o -e main -o a.out

My Ubuntu Dockerfile is:

FROM ubuntu:latest

WORKDIR /app

COPY . .

SHELL ["/bin/bash", "-c"]

RUN apt-get update && apt-get install -y nasm curl build-essential bc

RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
ENV PATH="/root/.cargo/bin:${PATH}"

CMD ["/bin/bash", "bin/test"]

Which runs a custom bash script ( ./bin/test ) which executes the assembly.

Running ldd --version on my machine (where it works fine), gives:

ldd (Ubuntu GLIBC 2.36-0ubuntu4) 2.36

And running it on the local docker container (where it fais) gives:

ldd (Ubuntu GLIBC 2.35-0ubuntu3.1) 2.35

But I can't imagine a minor version update would just happen to fix this problem? Anyway, running apt install libc-bin says the package is up to date on both of them.

The repo for my project is here https://github.com/revers3ntropy/oxynium , though it has a lot of wrappers around that assembly.

Thanks so much if you can help me!

Answer 1

My only idea is that I'm doing something with undefined behavior

You are. In particular, x86_64 ABI requires that you maintain stack alignment on 16-byte boundary, and you are not doing that.

On a machine on which GLIBC is built with -msse2 , calling into libc routines with mis-aligned stack may crash.

Examine the crashing instruction (using x/i $pc GDB command) -- chances are it's a MOVDQA or similar with misaligned source or destination address.