Python list-like object buffered to file?

Question

Before I reinvent the wheel, does there exist in Python a module that provides a simplistic class similar to a list that buffers itself to a file as needed?

In particular, I would like to do something like this:

b = BufferedList(limit = 10000, tmp = '/tmp')
b.append(some_tuple)
b.append(some_tuple)
b.append(some_tuple)
# ad nauseam 

This would create a buffered list named b. If the memory consumption of b exceeded 10000K bytes, further appends would write to a temp file.

Simple iteration is all I really need from the class. Features like pop, del, sort, and so forth would be nice but are not mandatory. The list will primarily contain tuples of strings.

I am using 2.7.

This will be implemented on Windows Server 2003.

Edit: As pointed out obliquely by Davidmh in the comments, I was really going at this backwards for my needs .

Since I do not really need pop, del, sort and other list manipulation methods, the cleanest solution for my needs today is just to set an appropriately large value for buffer when calling open() and always write to file if I think that it might possibly be needed.

I will just add a flag to my SQL tasks. It will probably end up with something like this:

imp = SqlImportTask()
imp.src_cs = std_config.DB04_CS
imp.dst_cs = std_config.SQL07_CS
imp.sql = 'SELECT * FROM PurchaseOrderDetail'
imp.dst_table = 'dbo.PurchaseOrderDetail'
imp.use_tmp_file = True

Answer 1

在Python中，有一个名为Pickle的对象序列化库