Huffman encoding: how to write binary data in Python

Question

I have tried methods using the struct module, as shown by the lines commented out in my code, but it didn't work out. Basically I have two options: I can either write the binary data code by code (my code are sequences of bits of length varying from 3 to 13 bits), or convert the whole string of n characters (n=25000+ in this case) to binary data. But I don't know how to implement either methods. Code:

import heapq
import binascii
import struct

def createFrequencyTupleList(inputFile):
    frequencyDic = {}

    intputFile = open(inputFile, 'r')
    for line in intputFile:
        for char in line:
            if char in frequencyDic.keys():
                frequencyDic[char] += 1
            else:
                frequencyDic[char] = 1

    intputFile.close()
    tupleList = []
    for myKey in frequencyDic:
        tupleList.append((frequencyDic[myKey],myKey))
    return tupleList

def createHuffmanTree(frequencyList):
    heapq.heapify(frequencyList)
    n = len(frequencyList)
    for i in range(1,n):
        left = heapq.heappop(frequencyList)
        right = heapq.heappop(frequencyList)
        newNode = (left[0] + right[0], left, right)
        heapq.heappush(frequencyList, newNode)
    return frequencyList[0]

def printHuffmanTree(myTree, someCode,prefix=''):
    if len(myTree) == 2:
        someCode.append((myTree[1] + "@" + prefix))
    else:
        printHuffmanTree(myTree[1], someCode,prefix + '0')
        printHuffmanTree(myTree[2], someCode,prefix + '1')

def parseCode(char, myCode):
    for k in myCode:
        if char == k[0]:
            return k[2:]


if __name__ == '__main__':
    myList = createFrequencyTupleList('input')
    myHTree = createHuffmanTree(myList)
    myCode = []
    printHuffmanTree(myHTree, myCode)
    inputFile = open('input', 'r')
    outputFile = open('encoded_file2', "w+b")
    asciiString = ''
    n=0
    for line in inputFile:
        for char in line:
            #outputFile.write(parseCode(char, myCode))
            asciiString += parseCode(char, myCode)
            n += len(parseCode(char, myCode))
    #values = asciiString
    #print n
    #s = struct.Struct('25216s')
    #packed_data = s.pack(values)
    #print packed_data
    inputFile.close()
    #outputFile.write(packed_data)
    outputFile.close()

Answer 1

You're looking for this:

packed_data = ''.join(chr(int(asciiString[i:i+8], 2)) 
                         for i in range(0, len(asciiString), 8))

It will take 8 bits at a time from the asciiString , interpret it as an integer, and output the corresponding byte.

Your problem here is that this requires the length of asciiString to be a multiple of 8 bits to work correctly. If not, you'll insert zero bits before the last few real bits.

So you need to store the number of bits in the last byte somewhere, so you know to ignore those bits when you get them back, instead of interpreting them as zeros. You could try:

packed_data = chr(len(asciiString) % 8) + packed_data

Then when you read it back:

packed_input = coded_file.read()
last_byte_length, packed_input, last_byte = (packed_input[0], 
                                             packed_input[1:-1], 
                                             packed_input[-1])
if not last_byte_length: last_byte_length = 8
ascii_input = ''.join(chain((bin(ord(byte))[2:].zfill(8) for byte in packed_input),
                      tuple(bin(ord(last_byte))[2:].zfill(last_byte_length),)))
# OR
# ascii_input = ''.join(chain(('{0:0=8b}'.format(byte) for byte in packed_input),
#                       tuple(('{0:0=' + str(last_byte_length) + '8b}').format(last_byte),)))

Edit: You either need to strip '0b' from the strings returned by bin() or, on 2.6 or newer, preferably use the new, alternate versions I added that use string formatting instead of bin() , slicing, and zfill() .

Edit: Thanks eryksun, good to use chain to avoid making a copy of the ASCII string. Also, need to call ord(byte) in the bin() version.