如何优化从RAM到光盘的写入？

Question

I have some python code for reading data from RAM of an FPGA and writing it to disk on my computer. The code's runtime is 2.56sec. I need to bring it down to 2sec.

mem = device.getNode("udaq.readout_mem").readBlock(16384)
device.dispatch()
ram.append(mem)
ram.reverse()
memory = ram.pop() 
for j in range(16384):
    if 0 < j < 4096:
        f.write('0x%05x\t0x%08x\n' %(j, memory[j]))
    if 8192 < j < 12288:
        f.write('0x%05x\t0x%08x\n' %(j, memory[j]))

Answer 1

Your loop is very unefficient. You're literally iterating for nothing when values aren't in range. And you're spending a lot of time testing the indices.

Don't do one loop & 2 tests. Just create 2 loops without index tests (note that first index is skipped if we respect your tests:

for j in range(1,4096):
    f.write('0x%05x\t0x%08x\n' %(j, memory[j]))
for j in range(8193,12288):
    f.write('0x%05x\t0x%08x\n' %(j, memory[j]))

maybe more pythonic & more concise (& not using memory[j] so it has a chance to be faster):

import itertools
for start,end in ((1,4096),(8193,12288)):
    sl = itertools.islice(memory,start,end)
    for j,m in enumerate(sl,start):
        f.write('0x%05x\t0x%08x\n' %(j, m))

the outer loop saves the 2 loops (so if there are more offsets, just add them in the tuple list). The islice object creates a slice of the memory but no copies are made. It iterates without checking the indices each time for array out of bounds, so it can be faster. It has yet to be benched, but the writing to disk is probably taking a lot of time as well.

Answer 2

Jean-François Fabre's observations on the loops are very good, but we can go further. The code is performing around 8000 write operations, of constant size, and with nearly the same content. We can prepare a buffer to do that in one operation.

# Prepare buffer with static portions
addresses = list(range(1,4096)) + list(range(8193,12288))
dataoffset = 2+5+1+2
linelength = dataoffset+8+1
buf = bytearray(b"".join(b'0x%05x\t0x%08x\n'%(j,0)
                        for j in addresses))

# Later on, fill in data
for line,address in enumerate(addresses):
    offset = linelength*line+dataoffset
    buf[offset:offset+8] = b"%08x"%memory[address]
f.write(buf)

This means far fewer system calls. It's likely we can go even further by eg reading the memory as a buffer and using b2a_hex or similar rather than a string formatting per word. It might also make sense to precalculate the offsets rather than using enumerate.