如何从多个分布式进程中并行将行写入日志文件

Question

I tried boost::log but I'm not getting anything in the file at all. In fact the file is not even being created.

Again: The point is to have many client processes distributed over the network writing messages into the same file.

And I've no use for all these attributes and sinks and sources and filters -- I don't even know what they are intended for. In fact I would prefer a simple constructor and a streaming operator.

Here is the current code producing no output at all:

#include <boost/log/core.hpp>
#include <boost/log/trivial.hpp>
#include <boost/log/expressions.hpp>
#include <boost/log/sinks/text_file_backend.hpp>
#include <boost/log/utility/setup/file.hpp>
#include <boost/log/utility/setup/common_attributes.hpp>
#include <boost/log/sources/severity_logger.hpp>
#include <boost/log/sources/record_ostream.hpp>


static int initFileLogging(void)
{       namespace logging = boost::log;
        namespace src = boost::log::sources;
        namespace sinks = boost::log::sinks;
        namespace keywords = boost::log::keywords;
        logging::add_file_log("~/smc.log");
        logging::core::get()->set_filter(logging::trivial::severity >= logging::trivial::info);
        logging::add_common_attributes();
        return 0;
}
int main(int, char**)
{
        initFileLogging();

        namespace logging = boost::log;
        namespace src = boost::log::sources;
        using namespace logging::trivial;
        src::severity_logger< severity_level > lg;
        BOOST_LOG_SEV(lg, debug) << "test" << std::endl;
}

Answer 1

I tried boost::log but I'm not getting anything in the file at all. In fact the file is not even being created.

In the sample code piece, you set up a global filter that only passes log records with severity level trivial::info or higher, but the log record you emit has level trivial::debug , which is lower. The log record gets discarded, and since no log record gets to the file sink backend, the file is not created.

And I've no use for all these attributes and sinks and sources and filters

Apparently, you do, since you're using a severity level, which is an attribute, and a filter and a file sink. You probably will be using a different sink type as well in the final solution.

You should really read the Design section in the library documentation to better understand how the library works in order to use it efficiently.

How to write lines into a logfile from within multiple distributed processes in parallel

The point is to have many client processes distributed over the network writing messages into the same file.

If you truly have multiple processes on different machines in a network, then there is no built-in solution in Boost.Log. If you're working on a UNIX-like system, you most likely have access to a syslog service, and Boost.Log can generate syslog messages with a syslog sink backend . You could configure the syslog service on all your machines to transmit messages to a common server that will write them in a common log file. Here, the syslog service implements the network communication and file writing parts.

Alternatively, you could create a TCP iostream using Boost.ASIO and use it with the ostream sink backend in Boost.Log. The TCP iostream would have to connect to the common server that will receive the formatted log records and write them to a file. You will have to write the server yourself though.

If your multiple processes are going to run on the same machine (ie not distributed over the network) then you could use IPC message queue backend to pass log records to a common process that will be writing the log file. This backend uses IPC message queue that is implemented using shared memory and may be more efficient than sockets. You will have to implement the server process yourself in this case as well, there is an example in the docs .

If none of this works, you could always implement your own sink backend .

In any case, you can see a common theme in all suggestions. You need to have a common process that will be writing the log file, and the client processes that will pass their log records to that common server. The question is only which transport to use and how to implement it.

You may be tempted to avoid using message passing design and instead use file locking to synchronize concurrent accesses of multiple processes to a common file (which, presumably, would be mounted in a shared folder on each client machine). Firstly, I should note that Boost.Log does not implement file locking, so using file sinks available out of the box on a common file like that will not work as desired. Next, I would advise against such approach anyway because (a) file locking may not work reliably over a network share and (b) file locking creates a contention point across all your client processes, which will affect scalability.