How to read 5 bytes to a meanful uint64_t in C?

Question

I need to alloc an array of uint64_t[1e9] to count something, and I know the items are between (0,2^39). So I want to calloc 5*1e9 bytes for the array.

Then I found that, if I want to make the uint64_t meanful, it is difficult to by pass the byte order.

There should be 2 ways.

One is to check the endianness first, so that we can memcpy the 5 bytes to either first or last of the whole 8 bytes.

The other is to use 5 bitshift and then bit-or them together.

I think the former should be faster.

So, under GCC or libc or GNU system, is there any header file to indicate whether the current system is Little Endian or Big Endian ? I know x86_64 is Little Endian, but I don't like to write a unportable code.

Of course any other idears are welcomed.

Add:

I need use the array to count many strings use D-left hashing. I plan to use 21bit for key and 18bit for counting.

Answer 1

When you say "faster"... how often is this code executed? 5 times <<8 plus an | probably costs less than 100ns. So if that code is executed 10'000 times, that adds up to 1 (one) second.

If the code is executed less times and you need more than 1 second to implement an endian-clean solution, you're wasting everyone's time.

That said, the solution to figure out the endianess is simple:

int a = 1;
char * ptr = (char*)&a;
bool littleEndian = *ptr == 1;

Now all you need it a big endian machine and a couple of test cases to make sure your memcpy solution works. Note that you need to need to call memcpy five times in one of the two cases to reorder the bytes.

Or you could simply shift and or five times...

EDIT I guess I misunderstood your question a bit. You're saying that you want to use the lowest 5 bytes (=40 bits) of the uint64_t as a counter, yes?

So the operation will be executed many, many times. Again, memcpy is utterly useless. Let's take the number 0x12345678 (32bit). In memory, that looks like so:

0x12 0x34 0x56 0x78    big endian
0x78 0x56 0x34 0x12    little endian

As you can see, the bytes are swapped. So to convert between the two, you must either use bit-shifting or byte swapping. memcpy doesn't work.

But that doesn't actually matter since the CPU will do the decoding for you. All you have to do is to shift the bits in the right place.

 key = item & 0x1FFFFF
 count = (item >>> 21)

to read and

 item = count << 21 | key

to write. Now you just need to build the key from the five bytes and you're done:

 key = (((hash[0] << 8) | (hash[1]<<8)) | ....

EDIT 2

It seems you have an array of 40bit ints and you want to read/write that array.

I have two solutions: Using memcpy should work as long as the data isn't copied between CPUs of different endianess (read: when you save/load the data to/from disk). But the function call might be too slow for such a huge array.

The other solution is to use two arrays:

int lower[];
unit8_t upper[]

that is: Save the bits 33-40 in a second array. To read/write the values, one shift+ or is necessary.

Answer 2

If you treat numbers as numbers, and not an array of bytes, your code will be endianess-agnostic. Hence, I would go for the shift and or solution.

Having said that, I really didn't catch what you are trying to do? Do you really need one billion entries, each five bytes long? If the data you are sampling is sparse, you might get away with allocating far less memory.

Answer 3

Well, I just find the kernel headers come with <asm/byteorder.h> .

inline memcpy to a while(i<x+3){++*i=++*j} may still slower since cache operation is slower than registers.

another way for memcpy is:

union dat {
 uint64_t a;
 char b[8];
} d;