OpenTelemetry 不在 Prometheus 中显示指标

Question

我已经在 Kubernetes 中设置了 OpenTelemetry。 下面是我的配置。

exporters:
  logging: {}
extensions:
  health_check: {}
  memory_ballast: {}
processors:
  batch: {}
  memory_limiter:
    check_interval: 5s
    limit_mib: 819
    spike_limit_mib: 256
receivers:
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
      thrift_compact:
        endpoint: 0.0.0.0:6831
      thrift_http:
        endpoint: 0.0.0.0:14268
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  prometheus:
    config:
      scrape_configs:
      - job_name: opentelemetry-collector
        scrape_interval: 10s
        static_configs:
        - targets:
          - ${MY_POD_IP}:8888
  zipkin:
    endpoint: 0.0.0.0:9411
service:
  extensions:
  - health_check
  - memory_ballast
  pipelines:
    logs:
      exporters:
      - logging
      processors:
      - memory_limiter
      - batch
      receivers:
      - otlp
    metrics:
      exporters:
      - logging
      processors:
      - memory_limiter
      - batch
      receivers:
      - otlp
      - prometheus
    traces:
      exporters:
      - logging
      processors:
      - memory_limiter
      - batch
      receivers:
      - otlp
      - jaeger
      - zipkin
  telemetry:
    metrics:
      address: 0.0.0.0:8888

端点在 Prometheus 中显示为 up。 但它没有显示任何数据。 当我检查 OTEL 收集器日志时，它显示如下

我已经在 Prometheus 中手动添加了抓取配置。

scrape_configs：

  - job_name: 'otel-collector'
    scrape_interval: 10s
    static_configs:
     - targets: ['opentelemetry-collector.opentelemetry:8888']

所以在 OTEL collectore configmap 中，我还看到了 Prometheus scrape 配置。

  prometheus:
    config:
      scrape_configs:
      - job_name: opentelemetry-collector
        scrape_interval: 10s
        static_configs:
        - targets:
          - ${MY_POD_IP}:8888

--添加--新--

kubectl get all -n thanos
NAME                                        READY   STATUS    RESTARTS   AGE
pod/thanos-query-776688f499-pvm24           1/1     Running   0          14h
pod/thanos-query-frontend-5b55d44cc-b6qx5   1/1     Running   0          14h

NAME                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)              AGE
service/thanos-query            ClusterIP   10.0.112.105   <none>        9090/TCP,10901/TCP   14h
service/thanos-query-frontend   ClusterIP   10.0.223.246   <none>        9090/TCP             14h

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/thanos-query            1/1     1            1           14h
deployment.apps/thanos-query-frontend   1/1     1            1           14h

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/thanos-query-776688f499           1         1         1       14h
replicaset.apps/thanos-query-frontend-5b55d44cc   1         1         1       14h

--日志--

2022-06-07T07:20:49.852Z        error   exporterhelper/queued_retry.go:183      Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: remote write returned HTTP status 404 Not Found; err = <nil>: 404 page not found\n", "dropped_items": 18}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
        go.opentelemetry.io/collector@v0.51.0/exporter/exporterhelper/queued_retry.go:183
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
        go.opentelemetry.io/collector@v0.51.0/exporter/exporterhelper/metrics.go:132
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
        go.opentelemetry.io/collector@v0.51.0/exporter/exporterhelper/queued_retry_inmemory.go:118
go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
        go.opentelemetry.io/collector@v0.51.0/exporter/exporterhelper/internal/bounded_memory_queue.go:82
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
        go.opentelemetry.io/collector@v0.51.0/exporter/exporterhelper/internal/bounded_memory_queue.go:69

我在 OpenTelemetry 中配置了以下 URL

exporters:
 prometheusremotewrite:
   endpoint: "http://thanos-query-frontend.thanos:9090/api/v1/write"

Answer 1

您只配置了logging导出器，它仅通过 zap.Logger 将数据导出到控制台 = 它不会将数据写入普罗米修斯。

还配置prometheusremotewrite导出器并将其添加到度量管道。 简约示例：

receivers:
  prometheus:
    config:
      scrape_configs:
      - job_name: opentelemetry-collector
        scrape_interval: 10s
        static_configs:
        - targets: ['localhost:8888']

exporters:
  prometheusremotewrite:
    endpoint: <example: my-prometheus/api/v1/write>

service:
  pipelines:
    metrics:
      receivers:
        - prometheus
      exporters:
        - prometheusremotewrite
  telemetry:
    metrics:
      address: 0.0.0.0:8888
      level: basic

有关prometheusremotewrite导出器，请参阅文档： https ://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusremotewriteexporter

如果您想要一个用于 OpenTelemetry 收集器遥测指标的 Grafana 仪表板，请参阅https://grafana.com/grafana/dashboards/15983 。