Python：读/写大量小文件

Question

I have a large number (about 60K) of small files (<10kb) that I need to process, create a processed version for each file and after that upload the processed files to a ftp server, in less than 1 minute.
I'm using Python 3 with standard read/write function and the performance is really bad.
Do you know if Python have any library support this case? Or what can I do to get better performance?
Thank you so much!

Answer 1

remember that I/O operations are heavy , and writing or reading files have overhead.

Writing 1024 new files each one is 1kb would take much longer than writing a single new file of 1mb. Every file you open the system need's to find a place for it on your file system and write it's metadata.

So no matter what you are planning, your performance probably won't be great , but I do suggest you to make sure you are doing the best to have the minimum of operations (ie upload all the files to the FTP server, instead of open the connection for each file and upload it by itself).
I suggest you to post a snippet from your code so people could help you make sure the code is the most minimal it can be