简体繁体 English

记录到非阻塞命名管道？

[英]Logging to a non blocking named pipe?

原文 2013-02-27 21:45:08 8 2 linux/ bash/ logging/ pipe/ fifo

I have a question, and I could'nt find help anywhere on stackoverflow or the web. 我有一个问题，我无法在stackoverflow或网络上的任何地方找到帮助。

I have a program (celery distributed task queue) and I have multiple instances (workers) each having a logfile (celery_worker1.log, celery_worker2.log). 我有一个程序（芹菜分布式任务队列），我有多个实例（工人），每个实例都有一个日志文件（celery_worker1.log，celery_worker2.log）。

The important errors are stored to a database, but I like to tail these logs from time to time when running new operations to make sure everything is ok (the loglevel is lower). 重要的错误存储在数据库中，但我喜欢在运行新操作时不时地拖尾这些日志以确保一切正常（loglevel更低）。

My problem: these logs are taking a lot of disk space. 我的问题：这些日志占用了大量磁盘空间。 What I would like to do: be able to "watch" the logs (tail -f) only when I need it, without them taking a lot of space. 我想做什么：能够只在我需要时“观察”日志（tail -f），而不需要占用大量空间。

My ideas until now: 我的想法到现在为止：

outputing logs to stdout, not to a file: not possible here since I have many workers outputing to different files, but I want to tail them all at once (tail -f celery_worker*.log) 输出日志到stdout，而不是文件：这里不可能，因为我有很多工人输出到不同的文件，但我想一次拖尾它们（tail -f celery_worker * .log）
using logrotate: it is an "OK" solution for me. 使用logrotate：对我来说这是一个“OK”的解决方案。 I don't want this to be a daily task but would rather not put a minute crontab for this, and more, the server is not mine so that would mean some work on the admin-sys side 我不希望这是一个日常任务，但宁愿不为此花一分钟crontab，而且更多，服务器不是我的，所以这意味着在admin-sys方面的一些工作
using named pipes: it looked good at first sight but I didn't know that named pipes (linux FIFO) where blocking. 使用命名管道：它一见钟情，但我不知道命名管道（linux FIFO）阻塞。 Hence, when I don't tail -f ALL of the pipes at the same time, or when I just quit my tail, the writing operations from the logger are blocked. 因此，当我不同时尾随-f所有管道，或者当我刚退出尾部时，记录器的写入操作被阻止。

Is there a way to have a non-blocking named pipe, which would just throw to stdout when tailed, and throw to /dev/null when not? 有没有办法有一个非阻塞的命名管道，它会在尾部时抛出stdout，并在没有时抛出/ dev / null？

Or are there technical difficulties to such a type of pipe? 或者这种管道有技术难题吗？ If there are, what are they? 如果有，他们是什么？

Thank you for your answers! 谢谢您的回答！

2 个解决方案

Have each worker log to stdout, but connect each stdout to a utility that automatically spools and rotates logs based on size or time. 让每个工作人员登录到stdout，但将每个stdout连接到一个实用程序，该实用程序根据大小或时间自动调整和旋转日志。 multilog and svlogd are examples of such. multilog和svlogd就是这样的例子。 For those programs, you'd merely tail the "current" log file. 对于那些程序，您只需拖尾“当前”日志文件。

You're right that logrotate is not quite the right solution for the problem you have. 你是对的，logrotate不是解决问题的正确方法。

Named pipes won't work as you want. 命名管道将无法正常工作。 At best, your writers could fill up their pipes and then discard subsequent logs, which is the inverse of the behavior you want. 最好的情况是，你的作者可以填满他们的管道然后丢弃后续日志，这与你想要的行为相反。

You could try shared memory device man:shm_overview or perhaps a number of them. 您可以尝试共享内存设备man：shm_overview或者其中一些。 You need to organise them as circular buffers so they'd store last N kb of your log and whenever you read them with reader it will output everything to your console. 您需要将它们组织为循环缓冲区，以便它们存储日志的最后N kb，每当您使用阅读器读取它们时，它都会将所有内容输出到您的控制台。 This approach is adopted by busybox's syslog/logread suit (see logread.c ). busybox的syslog / logread诉讼采用了这种方法（参见logread.c ）。