在 Grafana 中组合来自 Loki 的多个日志行

Question

I have stream of http logs (fastify pino format) via Loki that look like:

[2022-07-25T16:59:40.796Z] INFO: incoming request {"req":{"method":"GET","url":"/api/v1/teams/6vYE9rpOPl/members","hostname":"forge.flowforge.loc","remoteAddress":"10.1.106.162","remotePort":38422},"reqId":"req-t6"}
[2022-07-25T16:59:40.810Z] INFO: request completed {"res":{"statusCode":200},"responseTime":13.292339086532593,"reqId":"req-t6"}

I'd like to display average response time by path, but I'm struggling to work out how to combine the 2 log lines correlated by the reqId to get the url and responseTime together.

I can extract and parse the json for the 2 lines separately but not together.

Answer 1

I don't think it's doable with Loki alone. For that you would need either:

• some kind of a join, which is not supported (see https://github.com/grafana/loki/issues/3567 ), or
• some kind of a group-by/aggregation which is supported but in a very limited way ( https://grafana.com/docs/loki/latest/logql/metric_queries/#log-range-aggregations )

One solution I can think of is using Grafana transformations :

• use LogQL pattern and line_format to extract the JSON part from the logs, so that the line field becomes a valid JSON. Do not parse the JSON in LogQL: the extracted fields wouldn't be recognized by the Grafana transformations later on.
• apply "extract fields" transformation in order to parse the JSON (the transformation is currently in an Alpha version)
• add merge transformation
• outer join by reqId
• group by reqId

Grafana transformations are pretty powerful but a bit non-intuitive too, so the solution would require some experimenting. The debug transformation functionality might be helpful.