Python on Linux: How to *immediately* move files to USB to avoid issues from OS not syncing and the USB suddenly being pulled

Question

I have a Python script using shutil to move files from an internal folder to any present USB drives automatically. Even after the script closes, it seems that the OS does not always actually move the files to the USB drive until sometime later. My research suggests the OS does this delay for wear leveling purposes, but I am not certain that's the real explanation. Is there something I can add (ideally to my Python script or tied to it) so that the write will actually be executed immediately? Perhaps a different way to move the file, some way to demand an immediate sync, verifying the files were really moved, or doing some follow-on action that would require the OS to truly execute the move?

To be clear, I am not talking about problems from pulling the USB while the write is actively happening. The files are in total not many KB in size and I am seeing the problem even when the drive is pulled 40+ seconds later. The drives aren't blinking. I am not seeing partial writes where some files are moved and others aren't that would suggest I'm interrupting it midway through. Even if this was the issue, I think my question is still a valid question.

Background: I am running Raspbian Buster desktop version and Python 3.7.3 on a Pi4. I want to run this Pi without keyboard or monitor so that no user intervention is required to pull the drives or put in replacement ones. I have to run the desktop despite there being no monitor or else the Pi will not automount USB drives, but that is a separate issue. I am a noob, so especially thorough explanations would be particularly appreciated.

Answer 1

As @tdelaney said in their comment, the thorough-to-the-point-of-overkill way is to call flush() on the file object, then os.fsync() on its file descriptor, which since Raspbian is based on the Linux kernel shouldn't return until the buffers are flushed.

But if you're using one of the shutil.copy * functions, you don't have a file object to call those on, so you can just call out to os.system("sync") .

But note that even if the kernel is dusting its hands and saying "yup, flushed those buffers", the data could still be working its way through the device driver internals and not yet physically written to the media. You can't get a 100% guarantee that it's all the way through unless you're manually driving the USB logic, but the next best thing is adding a command to unmount the file system.

The problem is that code running from the USB stick can't unmount the filesystem containing itself, so if your Python is on the stick, it can't do the unmount. You can do it manually from the command line, though. Just umount /path/to/usb/stick should do it.