Retrieving unsigned 16-bit numbers from what seems an 8-bit data stream in Java

Question

I am working on a MIDI application using SysEx with a AH Qu-16.

Meter values are signed dB values (from -xxx to about 0 or a little over), coded as fixed point 7Q8 offset 8000 format, stored as unsigned 16 bit numbers, (transmitted in “7-bit-ized” format in the Sysex).

The example given is:

7-bit-ized binary            00100000 01111100 00000000
Unpacks to 8-bit-ized binary 01111100 10000000
Equivalent to hexadecimal:   7C80 
Remove the offset:           (int16_t) 7C80 – (int16_t) 8000 = FC80
Float and scale:             (float) FC80 / 256.0f = -3.5dB

I cannot however get values that make sense. Transforming the 8 bytes in to 7 bytes does not seem the issue, but getting sensible values does, and I wonder if it has to do with the 8 bit and 16 bit values.

Eg I am getting very low values which do not make sense (eg putting a signal in approx 0dB, it gives -210, and -126 with no sound), but when putting in the example values it work. Am I doing something wrong in the conversion? I just have a feeling it is something stupid, and I am not that familiar with these operations.

I added a sample from the mixer as hex at the bottom.

This is my code:

   private void processMeterReadings(String hex) throws Exception {
        byte[] fullInput;
        fullInput = Hex.decodeHex(hex.toCharArray());

        int groups = (fullInput.length / 8);
        int rest = fullInput.length % 8;
        byte[] meterData;
        if (rest == 0) meterData = new byte[groups * 7];
        else {
            meterData = new byte[groups * 7 + (rest - 1)];
            groups++;
        }
        //Work in groups of 8 (7 bytes) for x groups.
        int nrBits = 8;
        for (int g = 0; g < groups; g++) {
            if (rest > 0 && g == groups - 1) {
                //Logger.debug("Last group? g: " + g + " of total " + groups);
                nrBits = rest;
            }

            for (int i = 0; i < nrBits - 1; i++) { 
                meterData[g * 7 + i] = fullInput[g * 8 + i + 1];
                if((fullInput[g * 8] >> (Math.abs((i+1) - 7)) & 1) == 1) meterData[g * 7 + i] = (byte)(meterData[g * 7 + i] | (1 << 7));
            }
        }

        // Now we need to get each value (of 2 bytes to 16 bits int)??
        int totalValues = meterData.length / 2;
        float[] values = new float[totalValues];
        for (int i = 0; i < totalValues; i++) {
            ByteBuffer wrapped = ByteBuffer.wrap(meterData, i * 2, 2); // big-endian by default
            int num = wrapped.getShort(); // 1
            int num2 = Integer.parseInt("8000", 16);
            values[i] = ((num - num2) / 256f);
        }

    }


    String hex = "407f26011201120100120112011201124a356d0d000d00010012011201120112000112011201123616700d000d003a430101120112011274003a011201127a692c0d000d00513d0100120112011201120001120112136c0d58000d00513d0112000112011201120101120112136c0d00200d00011201120100120112011201121201127a6d0d000d400001120112011200011201120112010512136c0d000d00003355011201120100123e4d011201120a096c0d000d00742a5b513d513d513d51614f513d011236146e0d000d0001120001120112011201001201120112176c280d000d00011201001201120112011204011201126d750d50000d0001120112000112011201120109120112766b0d00200d0001120112010012011201120112020112166c0d000d400001120112011200011201120112010512166c0d000d00000112011201120100120112011201120a166c0d000d000100120112011201120001120112011216146c0d000d00011200011201120112010012011201120100200d0001000112010012011201120112000112011201000d400001000112011200011201120112010112011201000d000001000112011201001201120112011202011201000d000100000112011201120001120112011201041201000d000100000112011201120100120112011201120801000d000100010012011201120112000112011201120110000d000100011200011201120112010012011201120100240d000100513d01021201120112513d0001120112176c0d58000d00513d01120401120112513d0101120112176c0d00300d003a4301120108120112513d0112020112176c0d000d6000513d01120112100112513d0112010512176c0d000d00000112011201120100120112011201120a076c0d000d000100120112011201120001120112011207146c0d000d00011200011201120112005012001201120100200d0001000112010112011201120012200012011201000d400001000112011200011201120112010112011201120d000001000112011201001201120112011202011201120d000100000112011201120001120112011201041201120d000100000112011201120100120112011201120801120d000100010012011201120112000112011201120110120d000100011200011201120112010012011201120112200d0001000112010012011201120112000112011201120d400001000112011200011201120112010112011201120d000501000112743a74283a513d513d5b4754743a513d7b7f010a000112743a743a51513d513d5b4774283a513d7b7f0100000112011201120100120112011201121001127b7f0100010012011201120112000112011201120120127b7f010001120001120112011201001201120112011200011201120112010012011201120112000112011201120100120112011201120001120112011201001201120112011200011201120112010012011201120112000112011201120100120112011201120001120112011201001201120112011200011201120112010012011201120112000112011201120100120112011201120001120112011201001201120112011200011201120112010012011201120112000112011201120108120112454a1b4d0001120112011201001201120112011200011201120112010012011201120112000112011201000100000112011201120001120112011201001201120112011200011201120112010012011201120100000100011201120100120112011201120001120112011201001201120112011200011201120112010000010001120112000112011201120100120112011201120001120112";

Answer 1

I don't know what's wrong with your code, but I'd like to suggest a more systematic way of writing something fairly complex like this.

Build up your operations from the bottom up, and write unit tests for them as you go.

Then you will have a much better idea of what part of your implementation is correct, and where problems may lie.

eg (an implementation of 7bitized decoding)

public class SysEx {
    static byte[] unpack7BitIzedData(byte[] in) {
        int groups = in.length / 8;
        int remainder = in.length % 8;

        byte[] out = new byte[groups * 7 + (remainder - 1)];
        for (int g = 0; g < groups; ++g) {
            convertGroup(in, g * 8, 8, out, g * 7);
        }
        if (remainder > 0) {
            convertGroup(in, groups * 8, remainder, out, groups * 7);
        }
        return out;
    }

    /**
     * @param in the buffer containing the 7bitized data
     * @param inStart the start location in the buffer of the group we are decoding, i.e. the index of the hi bits byte
     * @param length the number of bytes in the gruop, including the hi bits byte
     * @param out the buffer to store the decoded data in
     * @param outStart the start location in the out buffer for the decoded data
     */
    static void convertGroup(byte[] in, int inStart, int length, byte[] out, int outStart) {
        int hiBits = in[inStart];
        for (int i = 0; i < length-1; ++i) {
            out[outStart+i] = (byte)(in[inStart+1+i] | topBit(hiBits, i));
        }
    }

    /**
     * @param hiBits The byte containing high bits for a group
     * @param targetByteIndex The index into the group (starting at 0)
     * @return 0x80 if the hi bit is set for that byte of the group, otherwise 0
     */
    static int topBit(int hiBits, int targetByteIndex) {
        if ((hiBits & 0x80) != 0) {
            throw new RuntimeException();
        }
        return ((hiBits & (0x40 >> targetByteIndex)) != 0 ? 0x80 : 0x00);
    }
}

import org.hamcrest.Matchers;
import org.junit.jupiter.api.Test;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;

public class TestSysEx {
    @Test
    public void testTopBit() {
        checkTopBit(0x00, 0,0,0,0,0,0,0);
        checkTopBit(0x40, 0x80,0,0,0,0,0,0);
        checkTopBit(0x60, 0x80,0x80,0,0,0,0,0);
    }

    private void checkTopBit(int data, Integer... values) {
        int i = 0;
        for (Integer v : values) {
            assertThat(SysEx.topBit((byte)data, i++), is(v));
        }
    }

    @Test
    public void testConvertGroup() {
        byte[] in = {0b00100000, 0b01111100, 0b00000000};
        byte[] out = new byte[2];
        byte[] expected = {0b01111100, (byte)0b10000000};
        SysEx.convertGroup(in, 0, 3, out, 0);
        assertThat(out, is(expected));
    }

    @Test
    public void testUnpack7BitIzedData() {
        byte[] in = {0b00100000, 0b01111100, 0b00000000};
        byte[] expected = {0b01111100, (byte)0b10000000};
        assertThat(SysEx.unpack7BitIzedData(in), is(expected));
    }

    @Test
    public void testUnpack7BitIzedData_longer() {
        byte[] in = {0b01010101, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0b00101010, 0x08, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, (byte)0b01000000, 0x0F};
        byte[] expected = {(byte)0x81, 0x02, (byte)0x83, 0x04, (byte)0x85, 0x06, (byte)0x87,
                0x08, (byte)0x89, 0x0A, (byte)0x8B, 0x0C, (byte)0x8D, 0x0E, (byte)0x8F};
        assertThat(SysEx.unpack7BitIzedData(in), is(expected));
    }
}