压缩Java中的字节数组并在C中解压缩

Question

I currently have the following array in a Java program,

byte[] data = new byte[800];

and I'd like to compress it before sending it to a microcontroller over serial (115200 Baud). I would like to then decompress the array on the microcontroller in C. However, I'm not quite sure what the best way to do this is. Performance is an issue since the microcontroller is just an arduino so it can't be too memory/cpu intensive. The data is more or less random ( edit I guess it's not really that random, see the edit below) I'd say since it represents a rgb color value for every 16 bits.

What would be the best way to compress this data? Any idea how much compression I could possibly get?

edit

Sorry about the lack of info. I need the compression to be lossless and I do only intend to send 800 bytes at a time. My issue is that 800 bytes won't transfer fast enough at the rate of 115200 baud that I am using. I was hoping I could shrink the size a little bit to improve speed.

Every two bytes looks like:

0RRRRRGGGGGBBBBB

Where RG and B bits represent the values for color channels red, green, and blue respectively. Every two bytes is then an individual LED on a 20x20 grid. I would imagine that many sets of two bytes would be identical since I frequently assign the same color codes to multiple LEDs. It may also be the case that RGB values are often > 15 since I typically use bright colors when I can (However, this might be a moot point since they are not all typically > 15 at once).

Answer 1

If the data is "more or less random" then you won't have much luck compressing it, I'm afraid.

UPDATE

Given the new information, I bet you don't need 32k colours on your LED display. I'd imagine that a 1024- or 256-colour palette might be sufficient. Hence you could get away with a trivial compression scheme (simply map each word through a lookup table, or possibly just discard lsbs of each component), that would work even for completely uncorrelated pixel values.

Answer 2

Use miniLZO compression. Java version C version

Answer 3

A really simple compression/decompression algorithm that is practical in tiny embedded environments and is easy to "roll your own" is run length encoding. Basically this means replacing a run of duplicate values with a (count, value) pair. Of course you need a sentinel (magic) value to introduce the pair, and then a mechanism to allow the magic value to appear in normal data (typically an escape sequence can be used for both jobs). In your example it might be best to use 16 bit values (2 bytes).

But naturally it all depends on the data. Data that is sufficiently random is incompressible by definition. You would do best to collect some example data first, then evaluate your compression options.

Edit after extra information posted

Just for fun and to show how easy run length encoding is I have coded up something. I'm afraid I've used C for compression as well, since I'm not a Java guy. To keep things simple I've worked entirely with 16 bit data. An optimization would be to use an 8 bit count in the (count,value) pair. I haven't tried to compile or test this code. See also my comment to your question about the possible benefits of mangling the LED addresses.

#define NBR_16BIT_WORDS 400
typedef unsigned short uint16_t;

// Return number of words written to dst (always
//  less than or equal to NBR_16BIT_WORDS)
uint16_t compress( uint16_t *src, uint16_t *dst )
{
    uint16_t *end = (src+NBR_16BIT_WORDS);
    uint16_t *dst_begin = dst;
    while( src < end )
    {
        uint16_t *temp;
        uint16_t count=1;
        for( temp=src+1; temp<end; temp++ )
        {
            if( *src == *temp )
                count++;
            else
                break;
        }
        if( count < 3 )
            *dst++ = *src++;
        else
        {
            *dst++ = (*src)|0x8000;
            *dst++ = count;
            *src += count;
        }
    }  
    return dst-dst_begin;
}

void decompress( uint16_t *src, uint16_t *dst )
{
    uint16_t *end_src = (src+NBR_16BIT_WORDS);
    uint16_t *end_dst = (dst+NBR_16BIT_WORDS);
    while( src<end_src && dst<end_dst )
    {
        data = *src++;
        if( (data&0x8000) == 0 )
            *dst++ = data;
        else
        {
            data  &= 0x7fff;
            uint16_t count = *src++;
            while( dst<end_dst && count-- )
                *dst++ = data;
        }
    }
}

Answer 4

One of the first things to do would be to convert from RGB to YUV, or YCrCb, or something on that order. Having done that, you can usually get away with sub-sampling the U and V (or Cr/Cb) channels to half resolution. This is quite common in most types of images (eg, JPEG, and MPEG both do it, and so do the sensors in most digital cameras).

Realistically, starting with only 800 bytes of data, most other forms of compression are going to be a waste of time and effort. You're going to have to put in quite a bit of work before you accomplish much (and keeping it reasonably fast on a Arduino won't be trivial for either).

Edit: okay, if you're absolutely certain you can't modify the data at all, things get more difficult very quickly. The real question at that point is what kind of input you're dealing with. Others have already mentioned the possibility of something on the order of a predictive delta compression -- eg, based on preceding pixels, predict what the next one is likely to be, and then encode only the difference between the prediction and the actual value. Getting the most out of that, however, generally requires running the result through some sort of entropy-based algorithm like Shannon-Fanno or Huffman compression. Those, unfortunately, aren't usually the fastest to decompress though.

If your data is most things like charts or graphs, where you can expect to have large areas of identical pixels, run-length (or run-end) encoding can work pretty well. This does have the advantage of being really trivial to decompress as well.

I doubt that LZ-based compression is going to work so well though. LZ-based compression works (in general) by building a dictionary of strings of bytes that have been seen, and when/if the same string of bytes is seen again, transmitting the code assigned to the previous instance instead of re-transmitting the entire string. The problem is that you can't transmit uncompressed bytes -- you start out by sending the code word that represents that byte in the dictionary. In your case, you could use (for example) a 10-bit code word. This means the first time you send any particularly character, you need to send it as 10 bits, not just 8. You only start to get some compression when you can build up some longer (two-byte, three-byte, etc.) strings in your dictionary, and find a matching string later in the input.

This means LZ-based compression usually gets fairly poor compression for the first couple hundred bytes or so, then about breaks even for a while, and only after it's been running across some input for a while does it really start to compress well. Dealing with only 800 bytes at a time, I'm not at all sure you're ever going to see much compression -- in fact, working in such small blocks, it wouldn't be particularly surprising to see the data expand on a fairly regular basis (especially if it's very random).

Answer 5

The data is more or less random I'd say since it represents a rgb color value for every 16 bits.

What would be the best way to compress this data? Any idea how much compression I could possibly get?

Ideally you can compress 800 bytes of colour data to one byte if the whole image is the same colour. As Oli Charlesworth mentions however, the more random the data, the less you can compress it. If your images looks like static on a TV, then indeed, good luck getting any compression out of it.

Answer 6

Definitely consider Oli Charlesworth's answer. On a 20x20 grid, I don't know if you need a full 32k color palette.

Also, in your earlier question , you said you were trying to run this on a 20ms period (50 Hz). Do you really need that much speed for this display? At 115200 bps, you can transmit ~11520 bytes/sec - call it 10KBps for a margin of safety (eg your micro might have a delay between bytes, you should do some experiments to see what the 'real' bandwidth is). At 50 Hz, this only allows you about 200 bytes per packet - you're looking for a compression ratio over 75%, which may not be attainable under any circumstances. You seem pretty married to your requirements, but it may be time for an awkward chat.

If you do want to go the compression route, you will probably just have to try several different algorithms with 'real' data, as others have said, and try different encodings. I bet you can find some extra processing time by doing matrix math, etc. in between receiving bytes over the serial link (you'll have about 80 microseconds between bytes) - if you use interrupts to read the serial data instead of polling, you can probably do pretty well by using a double buffer and processing/displaying the previous buffer while reading into the current buffer.

EDIT: Is it possible to increase the serial port speed beyond 115200? This USB-serial adapter at Amazon says it goes up to 1 Mbps (probably actually 921600 bps). Depending on your hardware and environment, you may have to worry about bad data, but if you increase the speed enough, you could probably add a checksum, and maybe even limited error correction.

I'm not familiar with the Arduino, but I've got an 8-bit FreeScale HCS08 I drive at 1.25 Mbps, although the bus is actually running RS-485, not RS-232 (485 uses differential signaling for better noise performance), and I don't have any problems with noise errors. You might even consider a USB RS-485 adapter, if you can wire that to your Arduino (you'd need conversion hardware to change the 485 signals to the Arduino's levels).

EDIT 2: You might also consider this USB-SPI/I2C adapter , if you have an available I2C or SPI interface, and you can handle the wiring. It says it can go to 400 kHz I2C or 200 kHz SPI, which is still not quite enough by itself, but you could split the data between the SPI/I2C and the serial link you already have.

Answer 7

LZ77/78 are relatively easy to write http://en.wikipedia.org/wiki/LZ77_and_LZ78

However given the small amount of data you're transferring, its probably not worth compressing it at all.