CreateProcessA function doesn't work in MASM64 (ml64.exe)

Question

I want to write program which starts another program (process). I am using MASM64 (ml64.exe) from Visual Studio 2015.

Program doesn't work. Nothing is shown. In debugger I get Access Violation.

I have no idea what is wrong with my code.

My code:

extrn ExitProcess : proc
extrn MessageBoxA : proc
extrn CreateProcessA : proc

PROCESS_INFORMATION    struct 
hProcess    DWORD    ?
hThread    DWORD    ?
dwProcessId    DWORD    ?
dwThreadId    DWORD    ?
PROCESS_INFORMATION    ends

STARTUPINFOA    struct 
cb    DWORD    ?
lpReserved    DWORD    ?
lpDesktop    DWORD    ?
lpTitle    DWORD    ?
dwX    DWORD    ?
dwY    DWORD    ?
dwXSize    DWORD    ?
dwYSize    DWORD    ?
dwXCountChars    DWORD    ?
dwYCountChars    DWORD    ?
dwFillAttribute    DWORD    ?
dwFlags    DWORD    ?
wShowWindow    WORD    ?
cbReserved2    WORD    ?
lpReserved2    DWORD    ?
hStdInput    DWORD    ?
hStdOutput    DWORD    ?
hStdError    DWORD    ?
STARTUPINFOA    ends

.const
MB_ICONINFORMATION equ 40h
ERROR_ALREADY_EXISTS equ 0B7h
NORMAL_PRIORITY_CLASS equ 020h

.data
szText db "This is first application which creates new process using CreateProcessA.", 00h
szCaption db "Information",00h
processInfo PROCESS_INFORMATION <>
startupInfo STARTUPINFOA <>
szProcName db "D:\Apps\SampleApp.exe", 00h

.code
    Main proc
        ;not sure if correct - begin
        lea rax, processInfo
        lea rbx, startupInfo


        sub rsp, 60h 
        push rax
        push rbx
        push 00h
        push 00h
        push NORMAL_PRIORITY_CLASS
        push 00h
        mov r9, 00h
        mov r8, 00h
        mov rdx, 00h
        lea rcx, szProcName
        call CreateProcessA
        add rsp, 60h 
        ;not sure if correct - end

        sub rsp, 28h
        mov r9, MB_ICONINFORMATION
        lea r8, szCaption
        lea rdx, szText
        xor rcx, rcx
        call MessageBoxA
        add rsp, 28h

        Exit:
        xor rcx, rcx
        call ExitProcess
    Main endp
end

build.bat

@echo off

ml64.exe prog1.asm /link /entry:Main /subsystem:windows /defaultlib:"kernel32.Lib" /defaultlib:"user32.Lib"

pause

Thanks in advance for Your help.

Answer 1

You should check definitions for PROCESS_INFORMATION and STARTUPINFO structures because they may differ between x86 and x64. For example, handles are defined as pointers, not DWORDs (32-bit integers).

Answer 2

Besides initializing STARTUPINFO and being sure to use the correct data types (pointers are 64-bits wide in 64-bit Windows, while DWORD is always 32-bits), you also need to correctly allocate the Parameter Area (also sometimes known as the "Shadow Space"):

The parameter area is always at the bottom of the stack (even if alloca is used), so that it will always be adjacent to the return address during any function call.
It contains at least four entries, but always enough space to hold all the parameters needed by any function that may be called.
Note that space is always allocated for the register parameters , even if the parameters themselves are never homed to the stack; a callee is guaranteed that space has been allocated for all its parameters.

^{Emphasis mine, from here}

Assuming a correct initialization of all the data structures required, a possible invocation of CreateProcessA is:

; Stack is assumed aligned here

push rax                     ; Not aligned (08h)
push rbx                     ; Aligned     (10h)
push 00h                     ; Not aligned (18h)                    
push 00h                     ; Aligned     (20h)
push NORMAL_PRIORITY_CLASS   ; Not aligned (28h)
push 00h                     ; Aligned     (30h)

; Make room for the first four (register) parameters
; Stack is aligned, not need to subtract 28h, just 20h (4*8)

sub rsp, 20h

mov r9, 00h                  
mov r8, 00h
mov rdx, 00h
lea rcx, szProcName
call CreateProcessA

add rsp, 50h

Note that the caller is responsible for cleaning up the stack after the call.

---

You idea of reserving space on the stack with a sub rsp, ... is not entirely wrong.
Of course you have to do the math correctly, but most importantly that technique is not immediately compatible with the push es.

Once you have sub rsp, ... one or more indirect stores of the kind mov [rsp+...], ... are required to set up the arguments. The push es will just move the stack pointer again , making all the previous work useless.

Answer 3

In your function reserve enough space for calling sub functions with so many Parameters. Use .ALLOCSTACK for that in your function prolog

Then simply assign the entire parameter list

mov QWORD PTR [rsp+48h], rax                     
mov QWORD PTR [rsp+40h], rbx                     
mov QWORD PTR [rsp+38h], 00h                     
mov QWORD PTR [rsp+30h], 00h                     
mov QWORD PTR [rsp+28h], NORMAL_PRIORITY_CLASS
mov QWORD PTR [rsp+20h], 00h

xor r9, r9    ; pass 0              
xor r8, r8    ; pass 0
xor edx, edx  ; pass 0  (higher DWORD becomes always also zero, saving the REX-byte)
lea rcx, szProcName
call CreateProcessA  

Answer 4

It's a bit old question but since I've just solved it writing it down. Now I was struggling with the same problem but with NASM. The issue is the same but the syntax will be a little bit different. Your problem is that your STARTUPINFOA and PROCESS_INFORMATION are incorrect for 2 reasons:

1. Pointers are DWORD64 on x64 system
2. You've not considered structure padding on 64bit systems

The correct structured in NASM syntax are here:

; https://msdn.microsoft.com/library/windows/desktop/ms686331.aspx
STRUC _STARTUPINFOA
    .cb:                resq 1
    .lpReserved:        resq 1
    .lpDesktop:         resq 1
    .lpTitle:           resq 1
    .dwX:               resd 1
    .dwY:               resd 1
    .dwXSize:           resd 1
    .dwYSize:           resd 1
    .dwXCountChars:     resd 1
    .dwYCountChars:     resd 1
    .dwFillAttribute:   resd 1
    .dwFlags:           resd 1
    .wShowWindow:       resd 1
    .cbReserved2:       resd 1
    .lpReserved2:       resq 1
    .hStdInput:         resq 1
    .hStdOutput:        resq 1
    .hStdError:         resq 1
ENDSTRUC

; https://msdn.microsoft.com/library/windows/desktop/ms684873.aspx
STRUC _PROCESS_INFORMATION
    .hProcess:          resq 1
    .hThread:           resq 1
    .dwProcessId:       resd 1
    .dwThreadId:        resd 1
ENDSTRUC

A little bit explanation:

1. resq = DWORD64
2. resd = DWORD32

If one follows the struct most of the fields can be filled properly but the are couple of fields which differ. The reason is the struct padding. MS compiler chooses the biggest element in the struct and then pads all other fields to it. In order to give an example STARTUPINFOA case looks like the following:

1. The compiler chooses DWORD64 as the biggest element
2. cb field is DWORD but since it can't pad it with the next field (lpReserved: DWORD64) it pads it to DWORD64
3. lpReserved, lpDesktop and lpTitle are already DWORD64
4. From dwX until dwFlags the size can be padded with the next element so no change
5. wShowWindow and cbReserved2 are only WORD so compiler pads them together to 8 bytes so each of the fields are changed to DWORD
6. From lpReserved2 until hStdError are already DWORD64

When I've done the correct padding it worked like charm. Good luck! :)