Systemd service and structured logging

Question

I have a systemd service which is of type simple .

I want my service to log key-value pairs.

Up to now my simple service just prints it log to stdout in a custom key-value syntax.

I would like to avoid my custom key-value syntax it use an official way to do structured logging.

Is there a way to use structured logs with systemd?

For example my service writes this to stdout:

{"key1": "value1", "key2": 1234}

It would be great of systemd could read the string as json.

Answer 1

You can log structured data to systemd and then view it as is or in json format

Let's say we log key-value pairs in json format

systemd-cat -t "struct_logs" echo '{"key1": "value1", "key2": 1234}'

Then we read them as is

sudo journalctl -t 'struct_logs' --lines 1 --no-pager

Apr 30 21:46:14 linux-ar struct_logs[17455]: {"key1": "value1", "key2": 1234}

Or in json format

sudo journalctl -t 'struct_logs' --lines 1 --no-pager -o json-pretty
{
        "__CURSOR" : "s=b6d07ffffffffff2787cea140a0db88a5;i=db8;b=c9b2132ffffffffe5807625fe;m=9fa5b3f81;t=58fffffff87eab;x=6402818ea7e32a5b",
        "__REALTIME_TIMESTAMP" : "1556671574343339",
        "__MONOTONIC_TIMESTAMP" : "42854989697",
        "_BOOT_ID" : "fffffffffffff24ec7a237b5e5807625fe",
        "PRIORITY" : "6",
        "_MACHINE_ID" : "fffffffffff66c86aaaaaaaaaa",
        "_HOSTNAME" : "linux-ar",
        "_TRANSPORT" : "stdout",
        "_UID" : "1000",
        "_GID" : "100",
        "SYSLOG_IDENTIFIER" : "struct_logs",
        "MESSAGE" : "{\"key1\": \"value1\", \"key2\": 1234}",
        "_PID" : "17455"
}

Going further you can recover your exact log messages with the help of jq

sudo journalctl -t 'struct_logs' --lines 1 --no-pager -o json-pretty | jq -r '.MESSAGE'
{"key1": "value1", "key2": 1234}

It could be possible also to log custom keys provided that the user has the proper rights to do that. A service should be able to do that

sudo logger --journald <<end
MESSAGE_ID=67feb6ffbaf24c5cbec13c008dd72309
MESSAGE=The dogs bark, but the caravan goes on.
SYSLOG_IDENTIFIER=struct_logs
KEY=bark
VALUE=goes on
end

Hope this helps.

Answer 2

Have a look at Structured logging to journald from within a docker container ; your application should use the sd_journal_send method if you want to specify custom fields.

If your application is unable to do that you can also output json - which will end up in the MESSAGE field you can then parse yourself. For example - I wrote SystemdJournal2Gelf to send the journal entries to Graylog and it also supports json output in the MESSAGE , sending it to Graylog as separate fields for filtering

If you don't want to modify the application you could use a separate script - something like this:

[Service]
ExecStart=/bin/sh -c '/bin/my-app | while read l; do echo "$l" | logger --journald; done'

You could even convert json to key/value pairs if you want, or use StandardOutput to send your standard-output to a socket where you have a separate service listening.

Answer 3

Well, systemd does not make it easy to do structured logging from text stream. It want your to take a dependencies on its library and use sd_journal_send (see for example http://0pointer.de/blog/projects/journal-submit.html ). I have not found any "text form" standard that allow systemd to understand that your line should not be smushed together in the MESSAGE fields but rather should be interpreted as a key-value pair. I guess a solution would be to write a helper process that parse your stdout and then call the systemd logger. You would then pipe your process stdout process into

However, I did found this answer that pipe directly into the journald socket. With clever use of jq, you could write something that exploit that. But that seems hacky: I do not know if systemd really intend to keep any compatibility on that socket.