What does this cast from long in a union to char* and back do?

Question

I was just examining the code of a simple executable packer which writes a section into the executable that unpacks it during startup, when I stumbled upon this piece of code:

void setDistance( unsigned long size )
{
    char* set = (((char *)I)+pUnpacker->VirtualAddress);
    union
    {
        short sh[2];
        long  l;
    } conv;
    conv.l      = size;
    conv.sh[0]  = 0;

    unpacker_set(set, (char *)(&conv.l), 4, TEXT_DISTANCE);
}

Size is the distance from the unpacker code in memory to the beginning of the Section that is supposed to be unpacked. In the loader code it is defined as a unsigned long. unpacker_set on the other hand has this code:

void inline unpacker_set( char* at, char* what, size_t size, unsigned long sig )
{
    DWORD oldprotect;
    unsigned char *set  = (unsigned char *)at;

    while(*((unsigned long*)(set)) != sig)
        set++;

    if(VirtualProtect(set, size, PAGE_READWRITE, &oldprotect) == TRUE)
        for(unsigned i=0; i<size; i++)
            *(set+i) = *(what+i);
}

Although I understand that second routine replaces the value from the unpacker code, but I would like to know why the hassle with a union is done. Any help would be appreciated.

Answer 1

Probably the best way to understand the code would be to write a very minimal test case and see what it does:

#include <iostream>

void f()
{
  union 
  {
    short sh[2];
    long l ;
  } conv ;
   conv.l = 100000000 ;

   std::cout << std::hex << conv.l << std::endl ;

  conv.sh[0] = 0 ;

  std::cout << std::hex << conv.l << std::endl ;
}

int main()
{
  f() ;
}

The output I see for this is as follows:

5f5e100
5f50000

So the code intention looks like it is trying to mask out the higher order bits of the size, although this is very ugly and it is unlikely to be portable.

As David pointed out you should be aware of strict aliasing . This article Type-punning and strict-aliasing is even better since it has some solid examples of real world issues using union s. So in order to assure that this code works as expected assuming gcc or clang you would need to pass in the following command line argument -fno-strict-aliasing .