如何调试 Telegraf 为什么不将数据从 Openweathermap 发送到 influxdb

Question

TL;DR: I am trying to setup TIG stack as follows, but listen udp :8125: bind: address already in use.

Openweather -> Telegraf -> InfluxDB -> Grafana

docker exec -it telegraf /bin/sh
# telegraf --config /etc/telegraf/telegraf.conf --once

output:

2022-06-22T13:09:27Z W! DeprecationWarning: Option "parse_data_dog_tags" of plugin "inputs.statsd" deprecated since version 1.10.0 and will be removed in 2.0.0: use 'datadog_extensions' instead
2022-06-22T13:09:27Z I! Starting Telegraf 1.23.0
2022-06-22T13:09:27Z I! Loaded inputs: cpu disk diskio kernel mem openweathermap processes statsd swap system
2022-06-22T13:09:27Z I! Loaded aggregators: 
2022-06-22T13:09:27Z I! Loaded processors: 
2022-06-22T13:09:27Z I! Loaded outputs: influxdb_v2 (2x)
2022-06-22T13:09:27Z I! Tags enabled: host=telegraf
2022-06-22T13:09:27Z W! Deprecated inputs: 0 and 1 options
2022-06-22T13:09:27Z E! [agent] Starting input inputs.statsd: listen udp :8125: bind: address already in use
2022-06-22T13:09:29Z I! [inputs.statsd] Stopping the statsd service
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2d3f999]

telegraf.conf

[global_tags]
[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  hostname = ""
  omit_hostname = false

[[outputs.influxdb_v2]]
  urls = ["http://influxdb:8086"]
  token = "$INFLUXDB_INIT_ADMIN_TOKEN"
  organization = "$INFLUXDB_INIT_ORG"
  bucket = "$INFLUXDB_INIT_BUCKET"
  namedrop = ["weather*"]
  
[[outputs.influxdb_v2]]
  urls = ["http://influxdb:8086"]
  token = "$INFLUXDB_OPENWEATHER_TOKEN"
  organization = "$INFLUXDB_INIT_ORG"
  bucket = "$INFLUXDB_OPENWEATHER_BUCKET"
  namepass = ["weather*"]
  
[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false
  report_active = false
[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "iso9660", "overlay", "aufs", "squashfs"]
[[inputs.diskio]]
[[inputs.kernel]]
[[inputs.mem]]
[[inputs.processes]]
[[inputs.swap]]
[[inputs.system]]

[[inputs.openweathermap]]
  app_id = "09b8953b9f8f891e15af01a369aeXXX"
  lang = "es"
  fetch = ["weather","forecast"]
  units = "metric"
  interval = "10m"
  
[[inputs.statsd]]
  protocol = "udp"
  max_tcp_connections = 250
  tcp_keep_alive = false
  service_address = ":8125"
  delete_gauges = true
  delete_counters = true
  delete_sets = true
  delete_timings = true
  percentiles = [50.0, 90.0, 99.0, 99.9, 99.95, 100.0]
  metric_separator = "_"
  parse_data_dog_tags = true
  datadog_extensions = false
  datadog_distributions = false
  allowed_pending_messages = 10000
  percentile_limit = 1000

docker-compose.yml

version: "3.9"

volumes:
  influxdb_data: {}
  postgres_data: {}
  grafana_data: {}
  

networks:
  influx:
    name: influx-frontend
  postgres:
    external: false

services:
  
  
  influxdb:
    container_name: influxdb
    image: influxdb:${INFLUXDB_VERSION}
    networks:
      - influx
    ports:
      - 8086:8086
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8086/ping"]
      interval: 10s
      timeout: 10s
      retries: 5
      start_period: 40s
    restart: always
    environment:
        - TZ=America/Argentina/Buenos_Aires
    env_file:
      - ./influxdb/influxdb.env
    volumes:
      - influxdb_data:/var/lib/influxdb2
      - ./influxdb/config.yml:/etc/influxdb2/config.yml
  
  telegraf:
    container_name: telegraf
    image: telegraf:${TELEGRAF_VERSION}
    hostname: telegraf
    profiles: ["telegraf"]
    environment:
        - TZ=America/Argentina/Buenos_Aires
    networks:
      - influx
    ports:
      - 8125:8125/udp
    restart: always
    depends_on:
      - influxdb
    env_file:
     - ./telegraf/telegraf.env
    volumes:
      - ./telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro

  postgres:
    container_name: postgres
    image: bitnami/postgresql:${POSTGRES_VERSION}
    profiles: ["grafana"]
    networks:
      - postgres
    ports:
      - 5432:5432
    restart: always
    env_file:
      - ./postgres/postgres.env
    volumes:
      - postgres_data:/bitnami/postgresql

  grafana:
    build:
      context: ./grafana
      dockerfile: Dockerfile
      args:
        - GRAFANA_VERSION=${GRAFANA_VERSION:-latest}
    container_name: grafana
    image: grafana/grafana-enterprise:${GRAFANA_VERSION}
    profiles: ["grafana"]
    networks:
      - postgres
      - influx
    ports:
      - 3000:3000
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/api/health"]
      interval: 10s
      timeout: 10s
      retries: 5
      start_period: 40s
    restart: always
    depends_on:
      - postgres
      - influxdb
    env_file:
      - ./grafana/grafana.env
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
      - ./grafana/dashboards:/etc/dashboards

In Influxdb database is created. I can access telegraf docker container and ping influxdb so it seems to be connected.

Would anyone be able to provide any guidance on how I can debug this to figure out why inputs.openweather data are not being sent by Telegraf to the database in Influxdb?

Thanks!

Answer 1

docker exec -it telegraf /bin/sh
# telegraf --config /etc/telegraf/telegraf.conf --once

This exec won't work, because in telegraf container there is already telegraf process running which occupies 8125 port, so that's why you get:

Starting input inputs.statsd: listen udp :8125: bind: address already in use

It is better option to debug at first option to check docker logs:

docker logs <container_id>

In these logs you should have information about first telegraf process in your docker container, which, I am almost sure, occupies port 8125.

You should also: check if docker exposes port 8125 data.

Then check with netstat if your machine accessed by Openweather have port 8125/UDP in listening status on a proper network interface.

It migh not be enough to fix your container but these things are basic for checking what is going on.