Traefik ingress to prometheus

Question

I would like to redirect incoming traffic to myserver.mydomain.com/prometheus to my prometheus pod. Here are my YAML files where I try to achieve this:

Here's the deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-server
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:latest
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
            - "--web.external-url=http://myserver.mydomain.com/prometheus"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf

        - name: prometheus-storage-volume
          persistentVolumeClaim:
            claimName: prometheus-local-zfs-pvc

the service manifest...

apiVersion: v1
kind: Service
metadata:
  name: prometheus-service
  namespace: monitoring
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '9090'
spec:
  selector:
    app: prometheus-server
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: 9090

...and the ingress manifest...

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: prometheus-ingress
  namespace: monitoring
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: myserver.mydomain.com
    http:
      paths:
      - path: /prometheus
        backend:
          serviceName: prometheus-service
          servicePort: 80

However, this returns a 404. What am I doing wrong?

EDIT: I have added the option as suggested by @coderanger. One thing I noticed when applying the new deployment was that there seemed to be a locking issue when the new pod is brought up before the old one is deleted, the error in the logs was err="opening storage failed: lock DB directory: resource temporarily unavailable....

Answer 1

You need to pass --web.external-url on the Prometheus command line options since you are moving it to a sub-path.