Promtail + Loki - Only shows some namespaces not all

Question

we recently decided to install loki and promtail via the loki-stack helm chart. Loki and promtail kind of work. We get some logs from Promtail and we can visualize them in grafana but our development namespace is nowhere to be found in loki. Promtail shows the development pod as an active target and promtail already collected the logs from the pod but we cant seem to get them into loki somehow... Any ideas?

Answer 1

tl;dr

set loki.monitoring.selfMonitoring.grafanaAgent.installOperator to false

---

This problem is caused by grafana-agent which is installed by default as a sub-chart of grafana/loki chart...

agent creates secret 'loki-logs-config' (loki in this case is Helm release name) which contains following configuration:

agent.yml: |+
  logs:
      configs:
          - clients:
              - external_labels:
                  cluster: loki
                url: http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push
            name: monitoring/loki
            scrape_configs:
              - job_name: podLogs/monitoring/loki
                kubernetes_sd_configs:
                  - namespaces:
                      names:
                          - monitoring
                    role: pod
                pipeline_stages:
                  - cri: {}
                relabel_configs:
                  - source_labels:
                      - job
                    target_label: __tmp_prometheus_job_name
                  - action: keep
                    regex: loki
                    source_labels:
                      - __meta_kubernetes_pod_label_app_kubernetes_io_instance
                  - action: keep
                    regex: loki
                    source_labels:
                      - __meta_kubernetes_pod_label_app_kubernetes_io_name
                  - source_labels:
                      - __meta_kubernetes_namespace
                    target_label: namespace
                  - source_labels:
                      - __meta_kubernetes_service_name
                    target_label: service
                  - source_labels:
                      - __meta_kubernetes_pod_name
                    target_label: pod
                  - source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - replacement: monitoring/loki
                    target_label: job
                  - replacement: /var/log/pods/*$1/*.log
                    separator: /
                    source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                    target_label: __path__
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_node_name
                    target_label: __host__
                  - action: labelmap
                    regex: __meta_kubernetes_pod_label_(.+)
                  - action: replace
                    replacement: monitoring/$1
                    source_labels:
                      - __meta_kubernetes_pod_controller_name
                    target_label: job
                  - action: replace
                    source_labels:
                      - __meta_kubernetes_pod_container_name
                    target_label: container
                  - action: replace
                    replacement: loki
                    target_label: cluster
      positions_directory: /var/lib/grafana-agent/data
  server: {}

As you can see under kubernetes_sd_configs there is namespaces list with value of monitoring - I have no idea why is it there, but that's the namespace I've installed this chart into. You won't see this secret after executing helm template - it seems that Grafana Agent creates it somehow after startup. It has label app.kubernetes.io/managed-by=grafana-agent-operator
Pretty magical if you ask me...

The solution for me was disabling disabling installation of Grafana Agent:

loki:
  loki:
    commonConfig:
      replication_factor: 1
    storage:
      type: 'filesystem'
    auth_enabled: false
  monitoring:
    dashboards:
      enabled: false
    selfMonitoring:
      enabled: true
      grafanaAgent:
        installOperator: false
      lokiCanary:
        enabled: false

Note: top loki element in the code block above is needed only if you add grafana/loki chart as subchart to your chart

---

IMO enabling beta feature (Grafana Agent is v0.30.0 today) in a Chart used as a reference in Loki's doc is insane:)