C++ - 即使在突然断电的情况下，如何将应用程序日志写入磁盘

Question

A short background:

I'm developing an embedded Linux application on a drone. In the last field test we had, the battery juice ran out without any warning during the flight. The drone had a rough landing. When I restarted it in the lab, out of 20 log files (*CSV and *TXT of different application modules), only 3 had data, and the rest were completely empty, but obviously, they had. to have data and lots of it, (14 minutes flight = 840 seconds, and at 50Hz frequency. we are expected to have 10ths of thousands of data lines in our logs).

So I have 2 questions:

1. Is it possible that while some of the logs flushed every few seconds (or less - maybe every write command), other logs are kept in a big buffer that is being flushed to memory only on a clean power down or when the buffer is getting exhausted?

2. Given that we are using C++, what is the best practice to make all logs being flushed more frequently to the disk (so in case of a sudden power down, I still have some logs to inspect), and still have as little as possible impact on performance?

thanks

Answer 1

The fundamental issue here is your program, your language, your operating system, and your hardware may each be buffering writes in order to delay them until the most opportune time.

Are you REALLY having performance problems with flushing each write? Or is that just theoretical?

Use one or more of the following recommendations as appropriate to your situation.

1. Optimize your logging. Do you really need to log that much? Remove debugging stuff that should only be in there when you are doing development, not when doing testing or production work. Wait for times when the processor isn't busy to flush queued logs. Use a separate thread for logging (threads add too much hassle to be worth the effort unless there is no other way around the problem). Maybe write to syslog instead? (I don't know if syslog has awesome log flushing capabilities or not). Since the drone is in contact with a base station or controller, can you just log over the wireless connection, and have the actual disk storage of the logs not even being done in the drone?
2. If you have actually tried flushing every write (or every now and then) and it slowed your application down too much, make sure there isn't some other junk running on your Linux environment that is stealing CPU or adding disk read/writes and that you aren't even using.
3. Always write a complete record (or multiple joined records) with a single function call. Don't write separate parts of the record with separate functions. If something fails you will end up with a partially written record. Combine all the elements of your record first, in memory, then write the entire thing, boom, done. Linux writes seem to be atomic. Once started, they will be fully done and nobody else will interrupt them BUT make sure you open the files in append mode to avoid confusion about where the writes will occur. With append mode, a write will always be at the end of the file, even if someone else slips a write in before you do. Multiple writers does not sound like a problem you have to deal with.
4. Don't buffer in your application for speed (you may have other reasons). That's old fashioned crap probably taught to you by a professor whose knowledge is outdated. Your operating system probably has a write buffer already. You don't need to do it yourself and in fact, that will cause a lot of problems if multiple processes need to share the file.
5. Don't use 50 year old C functions from the 1970s like fwrite or the iostreams >> abominations (they are just proofs of concept that somehow got adapted as a standard). They may do their own internal buffering. Use the operating system write() function. If your OS is Linux, do "man 2 write". Of course, you have to open and close the files with the other OS-level functions too. Do "man 2 open". Thre are all sorts of options to control disk caching etc. You can open a file with the attribute O_DIRECT meaning, "ask the operating system to minimize caching" and O_DSYNC which means your write won't return until it's been fully flushed to disk. This will bypass the OS level write buffer as much as possible. It's up to you and what you require.
6. There are "async" functions sometimes available (do "man aio") where you don't have to wait for the write to complete. The operating system will manage those for you and at least keep up the best it can. Even if it falls behind, as long as you write a full record at a time, it will minimize corruption if something goes wrong.
7. You want to use a "journaled" file system like Linux ext4 that doesn't actually append writes to the file until they are confirmed to have been fully written to disk. And if something goes wrong, when the disk is remounted, it will repair itself (as best it can).

I had to write an application with file integrity and I did need to buffer writes (because an event during processing could tell the application "don't write those queued up writes, we change our mind"). To avoid complications everything is combined into a single string then written with a single write() function when the time comes. I've never had a corrupt file even when the app was shutdown will kill -9. HOWEVER, it's never had power failures. (UPS backed up).

If you are working with a very underpowered platform, then a lot of the above may not be useful. Sorry, there wasn't enough detail in what you had to work with.