将python Web应用程序指标发送到InfluxDB

Question

For now I have a such monitoring system configured:

web_app (via python statsd client) -> statsd -> ...
    ... -> carbon-relay-ng -> carbon-cache -> whisper

And I use Grafana over Graphite as a graphing tool.

Due to too poor query performance I've decided to change this stack to just InfluxDB + Grafana bundle. So, my question is how can I send app metrics to the InfluxDB ? I prefer to keep this bundle pretty simple, so I would like to skip statsd if it's possible. Should I replace python statsd client with influxdb-python and use telegraf UDP service as an aggregation part in front of the InfluxDB or just send metrics directly to the InfluxDB instance?

Answer 1

I would send the data to telegraf using the line protocol.

I've used influxdb-python a lot to submit stats directly to InfluxDB. Sending the results locally to telegraf may well be quicker, depending on how quickly and reliably your InfluxDB installation responds - this will block your application if there are delays.

The line protocol seems easier to use to me than the other options, and telegraf can accept the line protocol directly. A potential downside is that anything you send that way would end up in the database allocated to telegraf stats. Going directly to InfluxDB you can choose which database your data will end up in, although it means bypassing the python module if you'd like to use the line protcol format.

To use influxdb-python and send to InfluxDB directly, you have a choice of JSON format or using a subclass of SeriesHelper

JSON

Creating the JSON structure that the write_points / write uses is really awkward and clunky. It only converts it into line format anyway.

Compare the JSON:

json_body = [
    {
        "measurement": "cpu_load_short",
        "tags": {
            "host": "server01",
            "region": "us-west"
        },
        "time": "2009-11-10T23:00:00Z",
        "fields": {
            "value": 0.64
        }
    }
]

to the line format:

# measurement,tag1=tag1value,tag2=tag2value column1=... 
cpu_load_short,host=server01,region=us-west value=0.64 1465290833288375000

I know which I think is easier to produce (And I know the timestamps don't match, I'm just using examples). The line format can be POST ed straight to InfluxDB using the requests library, or sent via UDP if that listener has been configured.

SeriesHelper

The module has a way to just accept the values and tags, by using SeriesHelper which can be awkward to set up, but is easy to use.

The example they give is:

from influxdb import InfluxDBClient, SeriesHelper

myclient = InfluxDBClient(host, port, user, password, dbname)

class MySeriesHelper(SeriesHelper):
    # Meta class stores time series helper configuration.
    class Meta:
        client = myclient
        series_name = 'events.stats.{server_name}'
        fields = ['some_stat', 'other_stat']
        tags = ['server_name']
        bulk_size = 5
        autocommit = True


MySeriesHelper(server_name='us.east-1', some_stat=159, other_stat=10)
MySeriesHelper(server_name='us.east-1', some_stat=158, other_stat=20)

So you can see from calling MySeriesHelper, that makes life easy once it's set up, but the configuration for the client either needs to be set up in the global scope (which is bad for a module) or in the class definition. This isn't good for getting configuration from a config file or service discovery, so you end up doing things like this in your config parsing functions:

# Read host, port, user password, dbname from config file, then:
MySeriesHelper.Meta.client = InfluxDBClient(host, port, user, password, dbname)
# Now it is safe to call MySeriesHelper

I've not had issues with reliability with influxdb-python, and most of the time we use SeriesHelper classes. It's not the most complex of things, but the idea behind metrics is not that one person with the knowledge adds it all, but that it's part of the way of life of all the people writing code at every part in the chain. From that perspective, ease of use is key to get people to adopt a tool.