Assembly - NASM Factorial issues

Question

Hi I am writing a factorial function in Assembly using NASM. I have to use russian multiplication in place of the mul for my assignment. I am using 32 bit linux

Here is my Factorial code

section .text
global factorial
extern rpmult

factorial:
    push    ebp
    mov     ebp, esp
    sub     esp, 4 ;creates memory for local variable at ebp-4
    mov     esi, [ebp+8] ; put n in esi
    cmp     esi, 1 ; n <= 1
    jbe     .done ; if so jump to done

.try:
    mov     [ebp-4],esi ;adds n temporarily into ebp-4
    dec     esi ; n - 1
    push    esi ; push arugment
    call    factorial ;call factorial again stores result in esi
    add     esp, 4 ;gets rid of the argument    

    mov     edi, esi ;copies n - 1 into edi    
    mov     esi,[ebp+4] ;gets the original value back (n)
    call    rpmult ;multiply
    jmp     .done ;once it reaches here, finished the function

.done:
    mov     esp, ebp ;restores esp
    pop     ebp
    ret     ;return the value

Here is my rpmult code:

section .text

global rpmult

rpmult:
    push    ebp
    mov     ebp, esp
    sub     esp, 4     ;allocate m

    mov     dword [ebp-4], 0   ; m = 0;
.while:
    test    edi, edi   ; x == 0?
    je      .done
    test    esi, esi   ; y == 0?
    je      .done

    test    edi, 0x01  ; x is odd?
    jz      .shifts
    add     [ebp-4], esi   ; m += y;

.shifts:
    shr     edi, 1     ; x >>= 1;
    shl     esi, 1     ; y <<= 1;
    jmp     .while

.done:
    mov     eax, [ebp-4]
    ;mov    esp, ebp
    ;pop    ebp
    leave
    ret

When I use the function through a C program, say the factorial of 4! I get

4! = 13803416593125867520

I believe my code is right, but I have no idea what to do. I need to get the factorial function working with the rpmult function for my final. Any help is appreciated! Thanks!

Answer 1

(Note: I've rewritten this answer, after looking at it again while I'm more awake, and having read @lloydm's comment.)

There are three problem areas:

1) The base case of the recursion

When debugging recursive functions, it's always sensible to check the base case first.

So what happens when calculating 1! ?

factorial:
push ebp
mov ebp, esp
sub esp, 4 ;creates memory for local variable at ebp-4
mov esi, [ebp+8] ; put n in esi
cmp esi, 1 ; n <= 1
jbe .done ; if so jump to done
...
.done:
mov esp, ebp ;restores esp
pop ebp
ret ;return the value

There are two issues here already:

1. You're expecting this code work correctly when called from C, which means that you need to follow the usual calling convention (which for Linux with gcc means "cdecl" - see http://en.wikipedia.org/wiki/X86_calling_conventions ). So you need to preserve esi , edi , ebp and ebx . But this code is overwriting whatever was in esi . This will lead to unpredictable behaviour when the function is called from C, because the code generated by the C compiler will assume that whatever was in esi before factorial was called is still there when it returns. You can only use these registers if you save their values somewhere first (and restore them before returning).

2. The return value is passed out in eax , but you're not putting anything into eax here. You want the answer for 1! to be 1 , not "whatever random junk happens to be in eax at the moment"!

2) The recursive case of the recursion

...
.try:
mov [ebp-4],esi ;adds n temporarily into ebp-4
dec esi ; n - 1
push esi ; push arugment
call factorial ;call factorial again stores result in esi
add esp, 4 ;gets rid of the argument    
mov edi, esi ;copies n - 1 into edi    
mov esi,[ebp+4] ;gets the original value back (n)
call rpmult ;multiply
jmp .done ;once it reaches here, finished the function
...

1. edi , like esi , is a register which needs to be preserved, as described above.

2. The line mov edi, esi ;copies n - 1 into edi is wrong. You don't want to put n - 1 into edi - you're trying to calculate (n-1)!*n here, so you want to put (n-1)! into edi , ie the answer calculated by the recursive call. Which, as @lloydm points out, is returned in eax . (I was misled by the comment in my original answer, and thought that you really were trying to put n - 1 into edi . That wouldn't work either because esi no longer contains n - 1 after the call factorial , because you don't follow the calling conventions.)

3. mov esi,[ebp+4] ;gets the original value back (n) is wrong (as I noted originally); [ebp+4] contains the return address; this should be [ebp-4] .

3) The strangely large value

4! = 13803416593125867520 4! = 13803416593125867520 is a stranger answer than it first appears: it's far too large for a 32-bit value. (In hex: 0xbf8f964200000000 , so it's a 64-bit value with a big number in the top 32 bits and zero in the bottom 32 bits.)

You might expect to get a completely random value as an answer, given the other bugs, but factorial returns a 32-bit random value. So why are you printing a 64-bit value here? (If you're not doing it deliberately, I suppose it could possibly be related to the C code doing something strange because esi and edi have not been preserved by your code.)

Debugging tip

Don't start by trying to work out why factorial(5) doesn't work. Start as simply as possible, with factorial(1) . Then work up to factorial(2) , etc.