How would I convert this crypto from C# to C

Question

This is the C# code I use:

public void Decrypt(byte[] @in, byte[] @out, int size)
{
    lock (this)
    {
        for (ushort i = 0; i < size; i++)
        {
            if (_server)
            {
                @out[i] = (byte)(@in[i] ^ 0xAB);
                @out[i] = (byte)((@out[i] << 4) | (@out[i] >> 4));
                @out[i] = (byte)(ConquerKeys.Key2[_inCounter >> 8] ^ @out[i]);
                @out[i] = (byte)(ConquerKeys.Key1[_inCounter & 0xFF] ^ @out[i]);
            }
            else
            {
                @out[i] = (byte)(ConquerKeys.Key1[_inCounter & 0xFF] ^ @in[i]);
                @out[i] = (byte)(ConquerKeys.Key2[_inCounter >> 8] ^ @out[i]);
                @out[i] = (byte)((@out[i] << 4) | (@out[i] >> 4));
                @out[i] = (byte)(@out[i] ^ 0xAB);
            }
            _inCounter = (ushort)(_inCounter + 1);
        }
    }
}

and this is how I converted it to work in C.

char* decrypt(char* in, int size, int server)
{
    char out[size];
    memset(out, 0, size);
    for (int i = 0; i < size; i++)
    {
        if (server == 1)
        {
            out[i] = in[i] ^ 0xAB;
            out[i] = out[i] << 4 | out[i] >> 4;
            out[i] = Key2[incounter >> 8] ^ out[i];
            out[i] = Key1[incounter & 0xFF] ^ in[i];
        }
        else if (server == 0)
        {
            out[i] = Key1[incounter & 0xFF] ^ in[i];
            out[i] = Key2[incounter >> 8] ^ out[i];
            out[i] = out[i] << 4 | out[i] >> 4;
            out[i] = out[i] ^ 0xAB;
        }
        incounter++;
    }
    return out;
}

However for some reason the C one does not work.

Link for the full C# file

Link for the full C file

Link for the C implementation

Answer 1

The most glaring error I see is that you are returning a pointer to a stack-allocated array, which is going to get stomped by the next function call after decrypt() returns. You need to malloc() that buffer or pass in a pointer to a writable buffer.

Answer 2

There was a translation error.

The C# line:

@out[i] = (byte)(ConquerKeys.Key1[_inCounter & 0xFF] ^ @out[i]);

Became:

out[i] = Key1[incounter & 0xFF] ^ in[i];

The value on the right of the xor (^) is from the wrong array.

Additionally, you are returning a stack-allocated variable, which will cause all sorts of problem.

Change:

char out[size];
memset(out, 0, size);

to:

char *out = (char*)calloc(size, sizeof(char));

Answer 3

You are returning a reference to a local variable which is illegal. Either let the caller pass in an array or use malloc() to create an array inside the method.

Answer 4

I also suggest turning char into unsigned char since it is more portable. If your platform assumes char is the same as signed char , the arithmetic (bit shifts, etc) will not work right.
So just specify unsigned char explicitly (use a typedef or include <stdint.h> if unsigned char seems too long-winded for you).