How to create a configurable python script to extract data for graphite?

Question

I have a software running on different machines (proxy, balancer, registrar...) and want to send the statistics to graphite using collectd .

For this purpose I have created a python script in which I make some regex searches on the return value of a command that lists all statistics belonging to this software. Since I want to use the same script on all different machines, I need to make it somehow configurable with a settings file. So that depending on the settings file, different parameters will be sent to graphite by each machine.

Part of my script:

stats = softwareX.get_all_statistics()   
call_mem = stats.findall(...mem_regex...).group(1)  
call_time = stats.findall(...time_regex...).group(1)  

Considering that I have many parameters extracted from stats and want to send only a certain group of them depending on the machine where the script runs. For expample, call_mem is one of the parameters to be sent to graphite by the balancer machine (but the call_time wont be sent) and it is the other way around for the proxy machine ( call_time will be sent, but not call_mem ).

How could this be done in a single configurable python script for all the machines?

Answer 1

I don't know if you still nedd an answer to this, but you could try something like this:

collect_and_send_stats.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys

all_stats_regex_dict = {
    'call_mem': 'here goes the call_mem regex',
    'call_time': 'here goes the call_time regex',
    # ...
}

def parse_stats(stat_name_list, all_stats_of_this_software):
    parsed_stats = {}
    for stat_name in stat_name_list:
        stat_regex = all_stats_regex_dict[stat_name]
        parsed_stats[stat_name] = all_stats_of_this_software.findall(stat_regex).group(1)

    return parsed_stats

if __name__ == '__main__':
    stat_name_list = sys.argv[1:]

    # somehow get the stats of this machine/software
    all_stats = get_all_statistics()

    stats = parse_stats(stat_name_list, all_stats)

    # send stats to graphite
    send(stats)

Then you could call this script like this on the balancer:

./collect_and_send_stats.py call_mem

and like this on the proxy

./collect_and_send_stats.py call_time

Hope this points you in a useful direction (if you haven't solved the problem yet).