简体繁体 English

如何将来自多个进程的日志写入按时间戳排序的单个文件？

[英]How to write logs from multiple processes to a single file sorted by timestamp?

原文 2020-05-20 05:07:35 9 1 linux/ sockets/ priority-queue/ mailboxer/ multiple-processes

Consider there are 3 processes that needs to log messages to a single file.考虑有 3 个进程需要将消息记录到单个文件中。 The rate of messages from the 3 processes are different.来自 3 个进程的消息速率不同。 Ex: P1 logs 10 requests per second;例如：P1 每秒记录 10 个请求； P2 logs 20 requests per second and P3 logs just 1 or 2 requests per second. P2 每秒记录 20 个请求，而 P3 每秒仅记录 1 或 2 个请求。 How can we ensure that these logs are written in the order of the timestamp?我们怎样才能保证这些日志是按照时间戳的顺序写的呢？ I mean it should be sorted by the timestamp before writing to the file.我的意思是它应该在写入文件之前按时间戳排序。

How could we achieve this?我们怎样才能做到这一点？

1 个解决方案

You could create a priority queue, ordered by time, that holds a few seconds' worth of messages.您可以创建一个按时间排序的优先级队列，其中包含几秒钟的消息。 When a processes calls the log service, the service attempts to add the message to the queue.当一个进程调用日志服务时，该服务会尝试将消息添加到队列中。 If the queue is full, then it pulls the first item (which is the oldest message) from the queue, outputs it, and then adds the new message to the queue.如果队列已满，则从队列中拉出第一项（即最旧的消息），将其输出，然后将新消息添加到队列中。

This allows for messages that come in slightly out of order, or for servers whose times are slightly off (by up to a few seconds).这允许消息的顺序稍有混乱，或者时间稍有偏差（最多几秒钟）的服务器。

You said that your processes log a total of 30 or 35 messages per second.您说您的进程每秒总共记录 30 或 35 条消息。 You probably want to allow maybe three times that to allow for traffic spikes, and then another two or three times that so you can hold two or three seconds of messages.您可能希望允许三倍以允许流量峰值，然后再允许两到三倍，以便您可以保留两到三秒的消息。 Let's just say the queue holds 300 messages.假设队列包含 300 条消息。 Make it 1,000 messages if you want to allow more leeway.如果您想留出更多余地，请将其设为 1,000 条消息。

And if you're worried about the queue not flushing fast enough during slow periods, you could either add a timer that flushes anything older than some set value, or make the writing code always remove anything that's older than some value.如果您担心队列在慢速期间刷新不够快，您可以添加一个计时器来刷新比某个设定值更早的任何内容，或者让编写代码始终删除任何比某个值更早的内容。

This isn't a perfect solution.这不是一个完美的解决方案。 If you suffer network delays, or if one of your servers' clocks is off by a lot, or some thread gets hung up for several seconds before calling the logging service, the message could get written to the log out of order, because newer messages were already written.如果您遇到网络延迟，或者如果您的一个服务器的时钟关闭了很多，或者某些线程在调用日志服务之前挂了几秒钟，则消息可能会被无序写入日志，因为更新的消息已经写好了。 You could identify such messages and handle them exceptionally.您可以识别此类消息并进行异常处理。 Of course, the larger your queue, the less likely it is for something like this to happen.当然，队列越大，发生这种情况的可能性就越小。

You might ask yourself it's really necessary that the logs be written in strict chronological order.您可能会问自己，确实有必要按照严格的时间顺序编写日志。