system() call returns “Permission Denied” on Windows XP

Question

I am experiencing a problem with a C program running on Windows XP that is getting Permission Denied (EACCES) errors when it tries to call system().

It doesn't seem to matter what I put in the command string, the commands all work manually but get Permission Denied errors when executed via system()

The other interesting thing is that the program works correctly on other XP machines, just not this one. That makes it feel like some kind of OS setting, but I am not totally sure what system() does under the hood and would like to understand what is happening here.

Here is my test code:

fprintf( stderr, "DEBUG: Running test of system(NULL) call...\n" );
if ( ( ret = system( NULL ) ) != 0)
    fprintf( stderr, "  DEBUG: ret: %d: errno: %d, %s\n", ret, errno, strerror( errno ) );

fprintf( stderr, "DEBUG: Running test of system(\"sleep 1\") call...\n" );
if ( ( ret = system( "sleep 1" ) ) != 0 )
    fprintf( stderr, "  DEBUG: ret: %d: errno: %d, %s\n", ret, errno, strerror( errno ) );

This produces an output of

DEBUG: Running test of system(NULL) call...
  DEBUG: ret: 1: errno: 0, No error
DEBUG: Running Test of system("sleep 1") call...
  DEBUG: ret: -1: errno: 13, Permission denied

Thanks.

UPDATE: I have also modified my code to use CreateProcess() instead of system(), and I still get an "Access Denied" error (5). This makes it sound even more like an OS setting, but I don't know where to look.

UPDATE2: Process Monitor shows the failure in a call to "QueryOpen" for the path "D:\\cmd.exe", which does not exist. All the other calls are for C:\\WINDOWS\\system32\\cmd.exe, so I don't know why the one call is bad.

Answer 1

Have you tried running your program with Process Monitor ? Process Monitor will show you all of the system calls your program makes, plus lots of other metadata about the calls (such as their arguments and their return values). It can help you debug why you're getting the "Permission Denied" error.

Answer 2

Are you sure you're checking for failure correctly? MSDN doesn't list EACCES as a possible failure. EACCES could simply be in errno from previous code.

int how_to_test_system(char const* command) {
  int rc;
  switch ((rc = system(command))) {
  case -1: // an error happened in calling system() itself
    perror("system"); // prints "system: <msg>"
    break;

  case 0: // command completed successfully
    break;

  default: // command executed, but had a non-zero return code
    // you might want to log something here
    // (otherwise you can simplify this switch into a single if for -1)
    break;
  }
  return rc;
}

Answer 3

COMSPEC environment variable is not set properly or is being overridden

Look for COMSPEC in 1. Built-in system variables

2. System variables found in the HKEY_LOCAL_MACHINE hive

3. Local variables found in the HKEY_CURRENT_USER hive

4. All environment variables and paths set in the Autoexec.bat file

5. All environment variables and paths set in a logon script (if present)

6. Variables used interactively in a script or batch file

The real meat of how this works is here... http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/ntcmds_shelloverview.mspx?mfr=true

Answer 4

Finally solved, the problem turned out to be a anti-virus service running in the background that was somehow interfering with things.

Thanks to everyone for the help.