整数的变长编码

Question

Whats the best way of doing variable length encoding of an unsigned integer value in C# ?

---

"The actual intent is to append a variable length encoded integer (bytes) to a file header."

For ex: "Content-Length" - Http Header

Can this be achieved with some changes in the logic below.

---

I have written some code which does that ....

Answer 1

A method I have used, which makes smaller values use fewer bytes, is to encode 7 bits of data + 1 bit of overhead pr. byte.

The encoding works only for positive values starting with zero, but can be modified if necessary to handle negative values as well.

The way the encoding works is like this:

• Grab the lowest 7 bits of your value and store them in a byte, this is what you're going to output
• Shift the value 7 bits to the right, getting rid of those 7 bits you just grabbed
• If the value is non-zero (ie. after you shifted away 7 bits from it), set the high bit of the byte you're going to output before you output it
• Output the byte
• If the value is non-zero (ie. same check that resulted in setting the high bit), go back and repeat the steps from the start

To decode:

• Start at bit-position 0
• Read one byte from the file
• Store whether the high bit is set, and mask it away
• OR in the rest of the byte into your final value, at the bit-position you're at
• If the high bit was set, increase the bit-position by 7, and repeat the steps, skipping the first one (don't reset the bit-position)

39    32 31    24 23    16 15     8 7      0
value:            |DDDDDDDD|CCCCCCCC|BBBBBBBB|AAAAAAAA|
encoded: |0000DDDD|xDDDDCCC|xCCCCCBB|xBBBBBBA|xAAAAAAA| (note, stored in reverse order)

As you can see, the encoded value might occupy one additional byte that is just half-way used, due to the overhead of the control bits. If you expand this to a 64-bit value, the additional byte will be completely used, so there will still only be one byte of extra overhead.

Note : Since the encoding stores values one byte at a time, always in the same order, big- or little-endian systems will not change the layout of this. The least significant byte is always stored first, etc.

Ranges and their encoded size:

0 -         127 : 1 byte
        128 -      16.383 : 2 bytes
     16.384 -   2.097.151 : 3 bytes
  2.097.152 - 268.435.455 : 4 bytes
268.435.456 -   max-int32 : 5 bytes

Here's C# implementations for both:

void Main()
{
    using (FileStream stream = new FileStream(@"c:\temp\test.dat", FileMode.Create))
    using (BinaryWriter writer = new BinaryWriter(stream))
        writer.EncodeInt32(123456789);

    using (FileStream stream = new FileStream(@"c:\temp\test.dat", FileMode.Open))
    using (BinaryReader reader = new BinaryReader(stream))
        reader.DecodeInt32().Dump();
}

// Define other methods and classes here

public static class Extensions
{
    /// <summary>
    /// Encodes the specified <see cref="Int32"/> value with a variable number of
    /// bytes, and writes the encoded bytes to the specified writer.
    /// </summary>
    /// <param name="writer">
    /// The <see cref="BinaryWriter"/> to write the encoded value to.
    /// </param>
    /// <param name="value">
    /// The <see cref="Int32"/> value to encode and write to the <paramref name="writer"/>.
    /// </param>
    /// <exception cref="ArgumentNullException">
    /// <para><paramref name="writer"/> is <c>null</c>.</para>
    /// </exception>
    /// <exception cref="ArgumentOutOfRangeException">
    /// <para><paramref name="value"/> is less than 0.</para>
    /// </exception>
    /// <remarks>
    /// See <see cref="DecodeInt32"/> for how to decode the value back from
    /// a <see cref="BinaryReader"/>.
    /// </remarks>
    public static void EncodeInt32(this BinaryWriter writer, int value)
    {
        if (writer == null)
            throw new ArgumentNullException("writer");
        if (value < 0)
            throw new ArgumentOutOfRangeException("value", value, "value must be 0 or greater");

        do
        {
            byte lower7bits = (byte)(value & 0x7f);
            value >>= 7;
            if (value > 0)
                lower7bits |= 128;
            writer.Write(lower7bits);
        } while (value > 0);
    }

    /// <summary>
    /// Decodes a <see cref="Int32"/> value from a variable number of
    /// bytes, originally encoded with <see cref="EncodeInt32"/> from the specified reader.
    /// </summary>
    /// <param name="reader">
    /// The <see cref="BinaryReader"/> to read the encoded value from.
    /// </param>
    /// <returns>
    /// The decoded <see cref="Int32"/> value.
    /// </returns>
    /// <exception cref="ArgumentNullException">
    /// <para><paramref name="reader"/> is <c>null</c>.</para>
    /// </exception>
    public static int DecodeInt32(this BinaryReader reader)
    {
        if (reader == null)
            throw new ArgumentNullException("reader");

        bool more = true;
        int value = 0;
        int shift = 0;
        while (more)
        {
            byte lower7bits = reader.ReadByte();
            more = (lower7bits & 128) != 0;
            value |= (lower7bits & 0x7f) << shift;
            shift += 7;
        }
        return value;
    }
}

Answer 2

You should first make an histogram of your value. If the distribution is random (that is, every bin of your histogram's count is close to the other), then you'll not be able encode more efficiently than the binary representation for this number.

If your histogram is unbalanced (that is, if some values are more present than others), then it might make sense to choose an encoding that's using less bits for these values, while using more bits for the other -unlikely- values.

For example, if the number you need to encode are 2x more likely to be smaller than 15 bits than larger, you can use the 16-th bit to tell so and only store/send 16 bits (if it's zero, then the upcoming byte will form a 16-bits numbers that can fit in a 32 bits number). If it's 1, then the upcoming 25 bits will form a 32 bits numbers. You loose one bit here but because it's unlikely, in the end, for a lot of number, you win more bits.

Obviously, this is a trivial case, and the extension of this to more than 2 cases is the Huffman algorithm that affect a "code word" that close-to optimum based on the probability of the numbers to appear.

There's also the arithmetic coding algorithm that does this too (and probably other).

In all cases, there is no solution that can store random value more efficiently than what's being done currently in computer memory.

You have to think about how long and how hard will be the implementation of such solution compared to the saving you'll get in the end to know if it's worth it. The language itself is not relevant here.

Answer 3

如果小值比大值更常见，您可以使用Golomb 编码。

Answer 4

I know this question was asked quite a few years ago, however for MIDI developers I thought to share some code from a personal midi project I'm working on. The code block is based on a segment from the book Maximum MIDI by Paul Messick (This example is a tweaked version for my own needs however, the concept is all there...).

    public struct VariableLength
    {
        // Variable Length byte array to int
        public VariableLength(byte[] bytes)
        {
            int index = 0;
            int value = 0;
            byte b;
            do
            {
                value = (value << 7) | ((b = bytes[index]) & 0x7F);
                index++;
            } while ((b & 0x80) != 0);

            Length = index;
            Value = value;
            Bytes = new byte[Length];
            Array.Copy(bytes, 0, Bytes, 0, Length);
        }

        // Variable Length int to byte array
        public VariableLength(int value)
        {
            Value = value;
            byte[] bytes = new byte[4];
            int index = 0;
            int buffer = value & 0x7F;

            while ((value >>= 7) > 0)
            {
                buffer <<= 8;
                buffer |= 0x80;
                buffer += (value & 0x7F);
            }
            while (true)
            {
                bytes[index] = (byte)buffer;
                index++;
                if ((buffer & 0x80) > 0)
                    buffer >>= 8;
                else
                    break;
            }

            Length = index;
            Bytes = new byte[index];
            Array.Copy(bytes, 0, Bytes, 0, Length);
        }

        // Number of bytes used to store the variable length value
        public int Length { get; private set; }
        // Variable Length Value
        public int Value { get; private set; }
        // Bytes representing the integer value
        public byte[] Bytes { get; private set; }
    }

How to use:

public void Example()
{   
//Convert an integer into a variable length byte
int varLenVal = 480;     
VariableLength v = new VariableLength(varLenVal);
byte[] bytes = v.Bytes;

//Convert a variable length byte array into an integer
byte[] varLenByte = new byte[2]{131, 96};     
VariableLength v = new VariableLength(varLenByte);
int result = v.Length;
}

Answer 5

BinaryReader.Read7BitEncodedInt Method ?

BinaryWriter.Write7BitEncodedInt Method ?

Answer 6

As Grimbly pointed out , there exists BinaryReader.Read7BitEncodedInt and BinaryWriter.Write7BitEncodedInt . However, these are internal methods that one cannot call from a BinaryReader or -Writer object.

However, what you can do is take the internal implementation and copy it from the reader and the writer :

public static int Read7BitEncodedInt(this BinaryReader br) {
    // Read out an Int32 7 bits at a time.  The high bit 
    // of the byte when on means to continue reading more bytes.
    int count = 0;
    int shift = 0;
    byte b;
    do {
        // Check for a corrupted stream.  Read a max of 5 bytes.
        // In a future version, add a DataFormatException.
        if (shift == 5 * 7)  // 5 bytes max per Int32, shift += 7
            throw new FormatException("Format_Bad7BitInt32");

        // ReadByte handles end of stream cases for us. 
        b = br.ReadByte();
        count |= (b & 0x7F) << shift;
        shift += 7;
    } while ((b & 0x80) != 0); 
    return count;
}   

public static void Write7BitEncodedInt(this BinaryWriter br, int value) {
    // Write out an int 7 bits at a time.  The high bit of the byte,
    // when on, tells reader to continue reading more bytes.
    uint v = (uint)value;   // support negative numbers
    while (v >= 0x80) {
        br.Write((byte)(v | 0x80));
        v >>= 7;
    }
    br.Write((byte)v);
}   

When you include this code in any class of your project, you'll be able to use the methods on any BinaryReader / BinaryWriter object. They've only been slightly modified to make them work outside of their original classes (for example by changing ReadByte() to br.ReadByte() ). The comments are from the original source.