Python多重处理：锁定适用于（大型）磁盘写入吗？

Question

I have multiprocessing code wherein each process does a disk write (pickling data), and the resulting pickle files can be upwards of 50 MB (and sometimes even more than 1 GB depending on what I'm doing). Also, different processes are not writing to the same file, each process writes a separate file (or set of files).

Would it be a good idea to implement a lock around disk writes so that only one process is writing to the disk at a time? Or would it be best to just let the operating system sort it out even if that means 4 processes may be trying to write 1 GB to the disk at the same time?

Answer 1

As long as the processes aren't fighting over the same file; let the OS sort it out. That's its job.

Unless your processes try and dump their data in one big write, the OS is in a better position to schedule disk writes. If you do use one big write, you mighy try and partition it in smaller chunks. That might give the OS a better chance of handling them.

Of course you will hit a limit somewhere. Your program might be the CPU-bound, memory-bound or disk-bound. It might hit different limits depending on the input or load. But unless you've got evidence that you're constantly disk-bound and you've got a good idea how to solve that, I'd say don't bother. Because the days that a write system call actuall meant that the data was directly sent to disk are long gone.

Most operating systems these days use unallocated RAM as a disk cache. And HDD's have built-in caches as well. Unless you disable both of these (which will give you a huge performance hit) there is precious little connection between your program completing a write and and the data actually hitting the plates or flash.

You might consider using memmap (if your OS supports it), and let the OS's virtual memory do the work for you. See eg the architect notes for the Varnish cache .