Read/Write CSV as binary with Python

Question

I just found out that I can save space\\ speed up reads of CSV files.

Using the answer of my previous question How do I create a CSV file from database in Python?

And 'wb' for opens

  w = csv.writer(open(Fn,'wb'),dialect='excel')

How can I open all files in a directory and saves all files with the same name as starting name and use 'wb' to reformat all files. I guess convert all CSV's to binary CSV's.

Answer 1

You can't "overwrite a file on the fly". You have two options:

1. if the files are small enough (smaller than the amount of available RAM by a comfortable margin), just loop over them ( os.listdir makes that loop easy, or os.walk if you want to catch the whole tree of subdirectories, not just one directory), and for each, read it in memory first, then overwrite the on-disk copy.

2. otherwise, loop over them, and each time write to a new file (eg by appending .new to the name), then move the new file over the old. This is safer (no risk of running out of memory, no risk of damaging a file if the computer crashes) but more complicated.

So, what is your situation: small-enough files (and backups for safeguard against computer and disk crashes), in which case I can if you wish show the simple code; or huge multi-GB files -- in which case it will have to be the complex code? Let us know!