Promtail labels

Question

I'm having trouble adding labels into Grafana, but this issue is only in one node. I have already 3 Promtails with labels working properly, I tried the same example on this machine which belongs to the same cluster and also has connectivity to Loki port.

Here is what I have:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /usr/hdp/promtail/data/positions.yaml

clients:
  - url: http://machine4:3100/loki/api/v1/push

scrape_configs:
  - job_name: zeppelin
    static_configs:
    - targets:
        - machine1:9080
      labels:
        host: machine1
        stream: zeppelin
        job: zeppelin
        __path__: /usr/hdp/logs/zeppelin/zeppelin-zeppelin-machine1.log
    pipeline_stages:
    - match:
        selector: '{job="zeppelin"}'
        stages:
        - regex:
            expression: '(?P<zeppelinError>RemoteInterpreterManagedProcess)'
        - labels:
            zeppelinError:

So when I go to grafana into variables I type 'label_values(zeppelinError) and don't show me the label.

Here are the logs from Promtail, with looks fine:

Aug 10 11:47:14 machine1 systemd[1]: Started Promtail service.
Aug 10 11:47:14 machine1 promtail[25496]: level=info ts=2021-08-10T10:47:14.666205865Z caller=server.go:225 http=0.0.0.0:9080 grpc=0.0.0.0:44903 msg="server listening on addresses"
Aug 10 11:47:14 machine1 promtail[25496]: level=info ts=2021-08-10T10:47:14.666573544Z caller=main.go:108 msg="Starting Promtail" version="(version=2.0.0, branch=HEAD, revision=6978ee5d)"
Aug 10 11:47:19 machine1 promtail[25496]: level=info ts=2021-08-10T10:47:19.663478261Z caller=filetargetmanager.go:261 msg="Adding target" key="{host=\"machine1\", job=\"zeppelin\", stream=\"zeppelin\"}"
Aug 10 11:47:19 machine1 promtail[25496]: level=info ts=2021-08-10T10:47:19.667623654Z caller=tailer.go:122 component=tailer msg="tail routine: started" path=/usr/hdp/logs/zeppelin/zeppelin-zeppelin-machine1.log
Aug 10 11:47:19 machine1 promtail[25496]: ts=2021-08-10T10:47:19.668843991Z caller=log.go:124 component=tailer level=info msg="Seeked /usr/hdp/logs/zeppelin/zeppelin-zeppelin-machine1.log - &{Offset:713999 Whence:0}"

And here the log I want to trace:

ERROR [2021-07-30 06:37:40,836] ({pool-4-thread-74} NotebookServer.java[afterStatusChange]:2294) - Error
java.lang.RuntimeException: 
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:205)

Probably is something small I'm missing here, hope you can give me a hand on this matter.

Following: https://grafana.com/docs/loki/latest/clients/promtail/stages/regex/#schema (how to capture data), https://github.com/google/re2/wiki/Syntax (Regex Expression Rules), https://sbcode.net/grafana/nginx-promtail/ (following a similar build)

Answer 1

Oddly, I ended up implementing this just like you did and had the same issue, so if we are doing something wrong, it means the docs are not as clear as they could be. I couldn't get more than one label to be recognised and the only working label would work on one machine but not another, with the same config.

Looking at the documentation though, I spotted that starting with Loki 2.3.0, LogQL supports dynamic creation of labels on query:

https://grafana.com/docs/loki/latest/logql/log_queries/#parser-expression

So, after spending way too long trying to get this fixed to no avail, I decided to rip out the pipeline stages from the Promtail config and apply the regex directly on the Loki query:

https://grafana.com/docs/loki/latest/logql/log_queries/#regular-expression

Which finally fixed my problem. I've now applied this on all Promtail-ed machines.

I hope this helps but if someone knows how to fix labelling via the Promtail config, that would still be very helpful to know as an alternative since there's a limit to dynamic labels before they start making querying too heavy.

Cheers.