Illegal assembly instruction in C

Question

I am trying to get generate an assembly instruction trace using a C program. I am using asm functions to insert assembly instructions which are illegal in x86_64 and detect them.

I have the following C program -

int main()
{
    asm(".byte 0x16"); // Illegal in x86_64
    asm(".byte 0x17"); // Illegal in x86_64
    return 0;
}

This is the instruction trace for the main function, which is expected -

.text:004004d6 55                               push   %rbp
.text:004004d7 48 89 e5                         mov    %rsp,%rbp
.text:004004da 16                               (bad)  
.text:004004db 17                               (bad)  
.text:004004dc b8 00 00 00 00                   mov    $0x0,%eax
.text:004004e1 5d                               pop    %rbp
.text:004004e2 c3                               retq   
.text:004004e3 66 2e 0f 1f 84 00 00 00 00 00    nopw  %cs:0x0(%rax,%rax,1)
.text:004004ed 0f 1f 00                         nopl   (%rax)

I am using Intel PIN to run the program trace and based on the instruction I am getting I do some kind of analysis. So for the instructions which are (bad), they are illegal and thus cause the program to crash. I check the instruction address for every instruction in the trace along with the address of the instruction which is next.

The next instruction address for the instruction mov %rsp, %rbp (address 0x004004d7, the one before the 0x16 instruction) as shown by Pin is 0x0. 0x0 is shown by Pin when the next instruction is a conditional/unconditional branch instruction. Since it is shown for the bad instructions as well, I suspect that the bad instructions are calling an exception handler which terminates the program and gives the following message -

Illegal instruction (core dumped)

I am using Pin to instrument the instructions and when I get the illegal instructions, I do some kind of simulation and then delete this instruction so that the trace can proceed further.

Is there any kind of setting/flags to disable this illegal instruction exception generation?

I am using gcc to generate my executable and it simple compiles without generating an error. The problem comes when I execute the instruction. Is there any setting that simply just walks over the instruction, when it encounters an illegal instruction?

Answer 1

As requested, here is the code I used to patch a ud2 undefined instruction to nop nop on unicore machines and nop lock on multicore machines. I've removed code that's not at all relevant to your use case:

void __gmon_start__(void)
{
   if(ill) return;
   ill=true;
   struct sigaction act;
   memset((void *) &act, 0, sizeof(act));
   act.sa_sigaction=sigill;
   act.sa_flags=SA_SIGINFO;
   sigaction(SIGILL, &act, NULL);
   // more code to set IsUP goes here
}

And:

static void sigill(int, struct siginfo *info, void *extra)
{
    // Dynamically patch a 'ud2' to a 'nop lock' or a 'nop nop'

    // Begin with a full CPU sync (purge unmodified code)
    __asm__ __volatile__("cpuid" : : : "ax", "bx", "cx", "dx");

 // There was code here to acquire a mutex to prevent
 // multiple attempts to change the same code

  ucontext_t *uc=(ucontext_t *) extra;
  if (uc==NULL) { char *j=NULL; *j++; } // fatal

  unsigned char *instr=(unsigned char *) uc->uc_mcontext.gregs[14]; // EIP
  if(instr==NULL) { char *j=NULL; *j++; } // fatal


    if(instr[0] != 0x90)
    { // instruction wasn't already patched

     if ( (instr[0] != 0x0f) || // must be a ud2
            ((instr[1] != 0x0b) && (instr[1] != 0xb9)) )
      { char *j=NULL; *j++; } // fatal

      mprotect( (void *) ( ((int) instr) & ~(PAGESIZE-1)),
           PAGESIZE, PROT_WRITE | PROT_EXEC | PROT_READ);
      instr[0] = 0x90; // first byte is always a nop
      instr[1] = (IsUP) ? 0x90 : 0xf0;
      mprotect( (void *) ( ((int) instr) & ~(PAGESIZE-1)),
          PAGESIZE, PROT_EXEC | PROT_READ);
   }

  // mutex was released here

  // Reflush the CPU
  __asm__ __volatile__("cpuid" : : : "ax", "bx", "cx", "dx");
 }

Sorry the code is a mess.