Convert floating point values to hexadecimal

Question

I'm struggling with a floating point to hex conversion in Lua. My application communicates with an old Akai S2000 sampler. The sampler codes two byte messages into four nibble values. The nibbles are in reverse order, hence the most significant nibble is last. There is one parameter that uses a binary encoding of the fraction part of the value. The two MS nibbles are used to encode the integral part of the value and the LS nibbles are used to encode the binary fraction.

Based on this discussion https://bytes.com/topic/c/answers/219928-how-convert-float-hex I have started to implement a Lua algorithm to generate these nibble values from a given parameter value. As I am not that strong with bit calculations I think I am doing many things the wrong way. There should be an easier way to compute these values and avoid a lot of my silly if/else hacks.

My code (pasted below) works for the positive numbers but it is a lot harder with the negative fractions.

In my tests I have added to tables with expected values. Each table works like this:

key = give value * 100 
value = expected outcome from my algorithm

Ie the first entry in the positiveNumbers table represents an input value of 0.01 and the expected output for that value is a four byte MemoryBlock containing 02 00 00 00 (The first two bytes represent the fraction and the last two the integral part).

Currently my algorithm fails at -0.94 and I can't hack around it without breaking for some other value.

Is there anybody who is string with bit calculations and that easily can see any noob mistake I have made especially with converting the negative values? Any help or pointers would be appreciated!

Lua Code:

function float2nibbles(value)
    local nibbles = MemoryBlock(4, true)

    -- Retreive integral and fraction parts of the given value to be converted
    local integ, fract = math.modf(math.abs(value))

    -- Calculate the values of the integral part (last two nibbles)
    local bi = BigInteger(integ)
    if value < 0 then
        -- This variable is sometimes added in the negative conversion of the MS nibbles
        local lsAdd = 1
        if integ == 0 then
            lsAdd = 0
        end
        nibbles:setByte(2, bit.band(bit.bnot(bi:getBitRangeAsInt(0,4)) + lsAdd, 0xF))
        nibbles:setByte(3, bit.band(bit.bnot(bi:getBitRangeAsInt(4,4)), 0xF))
    else
        nibbles:setByte(2, bit.band(bi:getBitRangeAsInt(0,4), 0xF))
        nibbles:setByte(3, bit.band(bi:getBitRangeAsInt(4,4), 0xF))
    end

    -- Calculate the values of the fraction (first two nibbles)
    local remainder = fract
    local prevRemain = 0
    for i = 1,2 do
        remainder = remainder * 16
        -- Integral part of the remainder
        local d = math.modf(remainder)
        if value < 0 and fract ~= 0 then
            local lsAdd = 1
            if fract == 0 or i == 1 then
                lsAdd = 0
            end
            console(string.format("lsAdd %d", lsAdd))
            nibbles:setByte(2 - i, bit.band(bit.bnot(d) + lsAdd, 0xF))
        else
            nibbles:setByte(2 - i, bit.band(d, 0xF))
        end
        console(string.format("fract %d = %d, %.2f", i, d, remainder))
        prevRemain = remainder
        remainder = remainder - d
    end

    -- For some reason this increment helps when the LS nibble should increment the value of the second nibble
    if nibbles:getByte(0) == 0 and nibbles:getByte(1) ~= 0 and value < 0 then
        console(string.format("weird increment { %d %d }", nibbles:getByte(0), nibbles:getByte(1)))
        nibbles:setByte(1, nibbles:getByte(1) + 1)
    end

    -- The precision of this data is one byte but apparently they seem to use a third increment to check for rounding
    remainder = remainder * 16
    console(string.format("final remainder %.2f", remainder))
    if math.abs(remainder - prevRemain) > 0.001 and remainder > 14 then
        console(string.format("overflow -> %.2f (%.2f)", remainder, prevRemain))
        if value < 0 then
            nibbles:setByte(0, nibbles:getByte(0) - 1)
        else
            nibbles:setByte(0, nibbles:getByte(0) + 1)
        end
    end

    console(string.format("%.2f : integral part %s (%s), fract %.2f", value, bit.tohex(integ, 2), nibbles:toHexString(1), fract))
    return nibbles
end

local positiveNumbers = {   
    "02 00 00 00",
    "05 00 00 00",
    "07 00 00 00",
    "0A 00 00 00",
    "0C 00 00 00",
    "0F 00 00 00",
    "02 01 00 00",
    "04 01 00 00",
    "07 01 00 00",
    "09 01 00 00",
    "0C 01 00 00",
    "0E 01 00 00",
    "01 02 00 00",
    "03 02 00 00",
    "06 02 00 00",
    "09 02 00 00",
    "0B 02 00 00",
    "0E 02 00 00",
    "00 03 00 00",
    "03 03 00 00",
    "05 03 00 00",
    "08 03 00 00",
    "0B 03 00 00",
    "0D 03 00 00",
    "00 04 00 00",
    "02 04 00 00",
    "05 04 00 00",
    "07 04 00 00",
    "0A 04 00 00",
    "0C 04 00 00",
    "0F 04 00 00",
    "02 05 00 00",
    "04 05 00 00",
    "07 05 00 00",
    "09 05 00 00",
    "0C 05 00 00",
    "0E 05 00 00",
    "01 06 00 00",
    "03 06 00 00",
    "06 06 00 00",
    "09 06 00 00",
    "0B 06 00 00",
    "0E 06 00 00",
    "00 07 00 00",
    "03 07 00 00",
    "05 07 00 00",
    "08 07 00 00",
    "0B 07 00 00",
    "0D 07 00 00",
    "00 08 00 00",
    "02 08 00 00",
    "05 08 00 00",
    "07 08 00 00",
    "0A 08 00 00",
    "0C 08 00 00",
    "0F 08 00 00",
    "02 09 00 00",
    "04 09 00 00",
    "07 09 00 00",
    "09 09 00 00",
    "0C 09 00 00",
    "0E 09 00 00",
    "01 0A 00 00",
    "03 0A 00 00",
    "06 0A 00 00",
    "09 0A 00 00",
    "0B 0A 00 00",
    "0E 0A 00 00",
    "00 0B 00 00",
    "03 0B 00 00",
    "05 0B 00 00",
    "08 0B 00 00",
    "0B 0B 00 00",
    "0D 0B 00 00",
    "00 0C 00 00",
    "02 0C 00 00",
    "05 0C 00 00",
    "07 0C 00 00",
    "0A 0C 00 00",
    "0C 0C 00 00",
    "0F 0C 00 00",
    "02 0D 00 00",
    "04 0D 00 00",
    "07 0D 00 00",
    "09 0D 00 00",
    "0C 0D 00 00",
    "0E 0D 00 00",
    "01 0E 00 00",
    "03 0E 00 00",
    "06 0E 00 00",
    "09 0E 00 00",
    "0B 0E 00 00",
    "0E 0E 00 00",
    "00 0F 00 00",
    "03 0F 00 00",
    "05 0F 00 00",
    "08 0F 00 00",
    "0B 0F 00 00",
    "0D 0F 00 00",
    "00 00 01 00"
}

local negativeNumbers = {
    "0E 0F 0F 0F",
    "0B 0F 0F 0F",
    "09 0F 0F 0F",
    "06 0F 0F 0F",
    "04 0F 0F 0F",
    "01 0F 0F 0F",
    "0E 0E 0F 0F",
    "0C 0E 0F 0F",
    "09 0E 0F 0F",
    "07 0E 0F 0F",
    "04 0E 0F 0F",
    "02 0E 0F 0F",
    "0F 0D 0F 0F",
    "0D 0D 0F 0F",
    "0A 0D 0F 0F",
    "07 0D 0F 0F",
    "05 0D 0F 0F",
    "02 0D 0F 0F",
    "00 0D 0F 0F",
    "0D 0C 0F 0F",
    "0B 0C 0F 0F",
    "08 0C 0F 0F",
    "05 0C 0F 0F",
    "03 0C 0F 0F",
    "00 0C 0F 0F",
    "0E 0B 0F 0F",
    "0B 0B 0F 0F",
    "09 0B 0F 0F",
    "06 0B 0F 0F",
    "04 0B 0F 0F",
    "01 0B 0F 0F",
    "0E 0A 0F 0F",
    "0C 0A 0F 0F",
    "09 0A 0F 0F",
    "07 0A 0F 0F",
    "04 0A 0F 0F",
    "02 0A 0F 0F",
    "0F 09 0F 0F",
    "0D 09 0F 0F",
    "0A 09 0F 0F",
    "07 09 0F 0F",
    "05 09 0F 0F",
    "02 09 0F 0F",
    "00 09 0F 0F",
    "0D 08 0F 0F",
    "0B 08 0F 0F",
    "08 08 0F 0F",
    "05 08 0F 0F",
    "03 08 0F 0F",
    "00 08 0F 0F",
    "0E 07 0F 0F",
    "0B 07 0F 0F",
    "09 07 0F 0F",
    "06 07 0F 0F",
    "04 07 0F 0F",
    "01 07 0F 0F",
    "0E 06 0F 0F",
    "0C 06 0F 0F",
    "09 06 0F 0F",
    "07 06 0F 0F",
    "04 06 0F 0F",
    "02 06 0F 0F",
    "0F 05 0F 0F",
    "0D 05 0F 0F",
    "0A 05 0F 0F",
    "07 05 0F 0F",
    "05 05 0F 0F",
    "02 05 0F 0F",
    "00 05 0F 0F",
    "0D 04 0F 0F",
    "0B 04 0F 0F",
    "08 04 0F 0F",
    "05 04 0F 0F",
    "03 04 0F 0F",
    "00 04 0F 0F",
    "0E 03 0F 0F",
    "0B 03 0F 0F",
    "09 03 0F 0F",
    "06 03 0F 0F",
    "04 03 0F 0F",
    "01 03 0F 0F",
    "0E 02 0F 0F",
    "0C 02 0F 0F",
    "09 02 0F 0F",
    "07 02 0F 0F",
    "04 02 0F 0F",
    "02 02 0F 0F",
    "0F 01 0F 0F",
    "0D 01 0F 0F",
    "0A 01 0F 0F",
    "07 01 0F 0F",
    "05 01 0F 0F",
    "02 01 0F 0F",
    "00 01 0F 0F",
    "0D 00 0F 0F",
    "0B 00 0F 0F",
    "08 00 0F 0F",
    "05 00 0F 0F",
    "03 00 0F 0F",
    "00 00 0F 0F"
}

function verifyFloat2Nibbles(value, expectedMemBlock)
    local temp = string.upper(float2nibbles(value):toHexString(1))
    assert(expectedMemBlock == temp, 
        string.format("Incorrect result for %.2f, expected %s, got %s", value, expectedMemBlock, temp))
end

for k,v in pairs(positiveNumbers) do
    verifyFloat2Nibbles(k / 100, v)
end

for k,v in pairs(negativeNumbers) do
    verifyFloat2Nibbles((k / 100) * -1, v)
end

Answer 1

One problem you have is that when you negate, you are not propagating the carry from one nibble to the next. The code is way too complicated to review and fix. Here's a redo:

n2a = {[0]='0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f'}

function f2h (f)
    -- TODO should range check f
    local scaledf = f * 256 -- move binary point 8 bits right
    -- TODO optionally you could also round scaledf here
    local scaledi = math.modf(scaledf) -- integer part
    -- convert to nibbles
    local scalediparts = { (0xf & scaledi),
                           (0xf & (scaledi >> 4)),
                           (0xf & (scaledi >> 8)),
                           (0xf & (scaledi >> 12)) }
    -- output in hex
    print(n2a[scalediparts[1]], n2a[scalediparts[2]], n2a[scalediparts[3]], n2a[scalediparts[4]])
    -- return the nibbles
    return scalediparts
end

and a console test:

> f2h(-0.01)
e   f   f   f
table: 0x7feaf9d06390
> f2h(-0.94)
0   1   f   f
table: 0x7feaf9e009a0
> f2h(0.01)
2   0   0   0
table: 0x7feaf9d06410
> 

This is using Lua 5.3 bit operators; conversion to the bit library is straightforward. You'll also need to modify the print statement and return value to match your preference.

Answer 2

function float2nibbles(value)
   local nibbles = MemoryBlock(4, true)
   local n = math.floor(math.abs(value)*256 + 0.13)
   n = value < 0 and 0x10000 - n or n
   for pos = 0, 3 do
      nibbles:setByte(pos, n%16)
      n = math.floor(n/16)
   end
   return nibbles
end