由于列表/数组太大，出现MemoryError错误

Question

Problem

I have to download object_x . For simplicity's sake, object_x comprises a series of integers adding up to 1000 . The download is irregular. I receive groups or chunks of integers in seemingly random order, and I need to keep track of them until I have all 1000 to make up the final object_x .

The incoming chunks can also overlap, so for instance:

Chunk 1: integers 0-500
Chunk 2: integers 600-1000
Chunk 3: integers 400-700

Current method

Create object_x as a list containing all of its comprising integers 0-1000 . When a chunk is downloaded, remove all of the integers that comprise the chunk from object_x . Keep doing this until object_x is empty (known to be complete then).

object_x = range(0,1000)

# download chunk 1
chunk = range(0, 500)

for number in chunk:
    if number in object_x:
        object_x.remove(number)

# repeat for every downloaded chunk

Conclusion

This method is very memory intensive. The script throws a MemoryError if object_x or chunk is too large.

I'm searching for a better way to keep track of the chunks to build the object_x . Any ideas? I'm using Python, but language doesn't matter I guess.

Answer 1

This is the kind of scenario where streaming is very important. Doing everything in memory is a bad idea because you might not have enough memory (as in your case). You should probably save the chunks to disk, keep track of how many you downloaded, and when you reach 1000, process them on disk (or load them into memory one by one to process them).

" C# Security: Computing File Hashes " is a recent article I wrote - it's a different subject, but it does illustrate the importance of streaming towards the end.