如何在C中连接字节数组

Question

My current concat function:

char* concat(char* a, int a_size,
             char* b, int b_size) {
    char* c = malloc(a_size + b_size);
    memcpy(c, a,            a_size);
    memcpy(c + a_size, b,   b_size);
    free(a);
    free(b);
    return c;
}

But this used extra memory. Is it possible to append two byte arrays using realloc without making extra memory space?

Like:

void append(char* a, int a_size, char* b, int b_size)
...

char* a = malloc(2);
char* b = malloc(2);

void append(a, 2, b, 2);
//The size of a will be 4.

Answer 1

While Jean-François Fabre answered the stated question, I'd like to point out that you can manage such byte arrays better by using a structure:

typedef struct {
    size_t         max;  /* Number of chars allocated for */
    size_t         len;  /* Number of chars in use */
    unsigned char *data;
} bytearray;
#define  BYTEARRAY_INIT  { 0, 0, NULL }

void bytearray_init(bytearray *barray)
{
    barray->max  = 0;
    barray->len  = 0;
    barray->data = NULL;
}

void bytearray_free(bytearray *barray)
{
    free(barray->data);
    barray->max  = 0;
    barray->len  = 0;
    barray->data = NULL;
}

To declare an empty byte array, you can use either bytearray myba = BYTEARRAY_INIT; or bytearray myba; bytearray_init(&myba); bytearray myba; bytearray_init(&myba); . The two are equivalent.

When you no longer need the array, call bytearray_free(&myba); . Note that free(NULL) is safe and does nothing, so it is perfectly safe to free a bytearray that you have initialized, but not used.

To append to a bytearray :

int bytearray_append(bytearray *barray, const void *from, const size_t size)
{
    if (barray->len + size > barray->max) {
        const size_t  len = barray->len + size;
        size_t        max;
        void         *data;

        /* Example policy: */
        if (len < 8)
            max = 8; /* At least 8 chars, */
        else
        if (len < 4194304)
            max = (3*len) / 2;  /* grow by 50% up to 4,194,304 bytes, */
        else
            max = (len | 2097151) + 2097153 - 24; /* then pad to next multiple of 2,097,152 sans 24 bytes. */

        data = realloc(barray->data, max);
        if (!data) {
            /* Not enough memory available. Old data is still valid. */
            return -1;
        }

        barray->max  = max;
        barray->data = data;
    }

    /* Copy appended data; we know there is room now. */
    memmove(barray->data + barray->len, from, size);
    barray->len += size;

    return 0;
}

Since this function can at least theoretically fail to reallocate memory, it will return 0 if successful, and nonzero if it cannot reallocate enough memory.

There is no need for a malloc() call, because realloc(NULL, size) is exactly equivalent to malloc(size) .

The "growth policy" is a very debatable issue. You can just make max = barray->len + size , and be done with it. However, dynamic memory management functions are relatively slow, so in practice, we don't want to call realloc() for every small little addition.

The above policy tries to do something better, but not too aggressive: it always allocates at least 8 characters, even if less is needed. Up to 4,194,304 characters, it allocates 50% extra. Above that, it rounds the allocation size to the next multiple of 2,097,152 and substracts 24. The reasoning behid this is complex, but it is more for illustration and understanding than anything else; it is definitely NOT "this is best, and this is what you should do too" . This policy ensures that each byte array allocates at most 4,194,304 = 2 ²² unused characters. However, 2,097,152 = 2 ²¹ is the size of a huge page on AMD64 (x86-64), and is a power-of-two multiple of a native page size on basically all architectures. It is also large enough to switch from so-called sbrk() allocation to memory mapping on basically all architectures that do that. It means that such huge allocations use a separate part of the heap for each, and the unused part is usually just virtual memory, not necessarily backed by any RAM, until accessed. As a result, this policy tends to work quite well for both very short byte arrays, and very long byte arrays, on most architectures.

Of course, if you know (or measure!) the typical size of the byte arrays in typical workloads, you can optimize the growth policy for that, and get even better results.

Finally, it uses memmove() instead of memcpy() , just in case someone wishes to repeat a part of the same byte array: memcpy() only works if the source and target areas do not overlap; memmove() works even in that case.

---

When using more advanced data structures, like hash tables, a variant of the above structure is often useful. (That is, this is much better in cases where you have lots of empty byte arrays.)

Instead of having a pointer to the data, the data is part of the structure itself, as a C99 flexible array member:

typedef struct {
    size_t         max;
    size_t         len;
    unsigned char  data[];
} bytearray;

You cannot declare a byte array itself (ie bytearray myba; will not work); you always declare a pointer to a such byte arrays: bytearray *myba = NULL; . The pointer being NULL is just treated the same as an empty byte array.

In particular, to see how many data items such an array has, you use an accessor function (also defined in the same header file as the data structure), rather than myba.len :

static inline size_t  bytearray_len(bytearray *const barray)
{
    return (barray) ? barray->len : 0;
}

static inline size_t  bytearray_max(bytearray *const barray)
{
    return (barray) ? barray->max : 0;
}

The (expression) ? (if-true) : (if-false) (expression) ? (if-true) : (if-false) is a ternary operator. In this case, the first function is exactly equivalent to

static inline size_t  bytearray_len(bytearray *const barray)
{
    if (barray)
        return barray->len;
    else
        return 0;
}

If you wonder about the bytearray *const barray , remember that pointer declarations are read from right to left, with * as "a pointer to". So, it just means that barray is constant, a pointer to a byte array. That is, we may change the data it points to, but we won't change the pointer itself. Compilers can usually detect such stuff themselves, but it may help; the main point is however to remind us human programmers that the pointer itself is not to be changed. (Such changes would only be visible within the function itself.)

Since such arrays often need to be resized, the resizing is often put into a separate helper function:

bytearray *bytearray_resize(bytearray *const barray, const size_t len)
{
    bytearray *temp;

    if (!len) {
        free(barray);
        errno = 0;
        return NULL;
    }

    if (!barray) {
        temp = malloc(sizeof (bytearray) + len * sizeof barray->data[0]);
        if (!temp) {
            errno = ENOMEM;
            return NULL;
        }

        temp->max = len;
        temp->len = 0;
        return temp;
    }

    if (barray->len > len)
        barray->len = len;

    if (barray->max == len)
        return barray;

    temp = realloc(barray, sizeof (bytearray) + len * sizeof barray->data[0]);
    if (!temp) {
        free(barray);
        errno = ENOMEM;
        return NULL;
    }

    temp->max = len;
    return temp;
}

What does that errno = 0 do in there? The idea is that because resizing/reallocating a byte array may change the pointer, we return the new one. If the allocation fails, we return NULL with errno == ENOMEM , just like malloc() / realloc() do. However, since the desired new length was zero, this saves memory by freeing the old byte array if any, and returns NULL . But since that is not an error, we set errno to zero, so that it is easier for callers to check if an error occurred or not. (If the function returns NULL , check errno . If errno is nonzero, an error occurred; you can use strerror(errno) to get a descriptive error message.)

You probably also noted the sizeof barray->data[0] , used even when barray is NULL. This is okay, because sizeof is not a function, but an operator: it does not access the right side at all, it only evaluates to the size of the thing the right side refers to. (You only need to use parentheses when the right size is a type.) This form is nice, because it lets a programmer change the type of the data member, without changing any other code.

To append data to such a byte array, we probably want to be able to specify whether we anticipate further appends to the same array, or whether this is probably the final append, so that only the exact needed amount of memory is needed. For simplicity, I'll only implement the exact size version here. Note that this function returns a pointer to the (modified) byte array:

bytearray *bytearray_append(bytearray *barray,
                            const void *from, const size_t size,
                            int exact)
{
    size_t  len = bytearray_len(barray) + size;

    if (exact) {
        barray = bytearray_resize(barray, len);
        if (!barray)
            return NULL; /* errno already set by bytearray_resize(). */

    } else
    if (bytearray_max(barray) < len) {            

        if (!exact) {

            /* Apply growth policy */
            if (len < 8)
                len = 8;
            else
            if (len < 4194304)
                len = (3 * len) / 2;
            else
                len = (len | 2097151) + 2097153 - 24;
        }

        barray = bytearray_resize(barray, len);
        if (!barray)
            return NULL; /* errno already set by the bytearray_resize() call */
    }

    if (size) {
        memmove(barray->data + barray->len, from, size);
        barray->len += size;
    }

    return barray;
}

This time, we declared bytearray *barray , because we change where barray points to in the function. If the fourth parameter, final , is nonzero, then the resulting byte array is exactly the size needed; otherwise the growth policy is applied.

Answer 2

yes, since realloc will preserve the start of your buffer if the new size is bigger:

char* concat(char* a, size_t a_size,
             char* b, size_t b_size) {
    char* c = realloc(a, a_size + b_size);
    memcpy(c + a_size, b,  b_size);  // dest is after "a" data, source is b with b_size
    free(b);
    return c;
}

c may be different from a (if the original memory block cannot be resized in-place contiguously to the new size by the system) but if that's the case, the location pointed by a will be freed (you must not free it), and the original data will be "moved".

My advice is to warn the users of your function that the input buffers must be allocated using malloc , else it will crash badly.