python, how to run commands on remote hosts and show output in GUI in real time?

Question

I know this is a real open-ended question, but I'm new to python and am building a simple one off web app to give a non technical team some self service capabilities. This team has a bunch of repetitive tasks that they kick over to another team that are just begging to be automated, like restarting a few processes on remote hosts, grep logs, cleanup old files, deploy/restart new versions of an application, get current running versions, etc. The users will be clicking buttons and watching the output in the GUI, they WILL NOT be manually entering commands to run (I know this is dangerous). Any new tasks will be scripted and added to the app from the technical support team.

Now, the only piece I'm not sure on is how to get (near) real time output from the commands back to the GUI. I've built a very similar app in PHP in the past, and what I did was flush the output of the remote commands to a db, and then would poll the db with ajax and append new output. It was pretty simple and worked great even though the output would come back in chunks (I had the output written to the GUI line by line, so it looked like it was real time). Is there a better way to do this? I was thinking of using web sockets to push the output of the command back to the GUI. Good idea? Bad idea? Anything better with a python library? I can also use nodejs, if that makes any difference, but I'm new to both languages (I do already have a simple python flask application up and running that acts as an API to glue together a few business applications, not a big deal to re-write in node).

Answer 1

This is a broad question, but I'll give you few clues.

Nice example is LogIo . Once you are willing to run some commands and than push output to GUI, using Node.js becomes natural approach. This app may contain few elements:

• part one that runs commands and harvests output and pushes it to
• part two that receives output and saves it to DB/files. After save, this part is throwing event to
• part three , that should be a websocket server, which will handle users that are online and distribute events to
• part four, which would be preoperly scripted GUI that is able to connect via websocket to part three, log-in user, receive events and broadcast them to other GUI elements.

Once I assume you feel stronger with PHP than python, for you easiest approach would be to create part two as a PHP service to handle input (save harvested output to db) and than, let say use UDP package to part three's UDP listening-socket.

Part one would be python script to just get command output and bypass it properly to part two . It should be as easy to hadle as usual grep case:

tail -f /var/log/apache2/access.log | /usr/share/bin/myharvester 

at some point of developing it you will be in demand of passing there also user or unical task id as parameter after myharvester .

Tricky but easier than you think will be to create a Node.js cript as part three . As a single instance script it should be able to receive input and bypass it to users as events. I've commited comething like this before:

var config = {};
var app = require('http').createServer().listen(config.server.port);

var io = require('socket.io').listen(app);

var listenerDgram = require('dgram').createSocket('udp4');
listenerDgram.bind(config.listeners.udp.port);

var sprintf = require('sprintf').sprintf;

var users = [];

app.on('error', function(er) {
    console.log(sprintf('[%s] [ERROR] HTTP Server at port %s has thrown %s', Date(), config.server.port, er.toString()));
    process.exit();
});

listenerDgram.on('error', function(er) {
    console.log(sprintf('[%s] [ERROR] UDP Listener at port %s has thrown %s', Date(), config.listeners.udp.port, er.toString()));
    process.exit();
});

listenerDgram.on('message', function(msg, rinfo) {
    // handling, let's say, JSONized msg from part two script,
    // buildinf a var frame and finally
    if(user) {
        // emit to single user based on what happened
        // inside this method
        users[user].emit('notification', frame);
    } else {
        // emit to all users
        io.emit('notification', frame);
    }

});

io.sockets.on('connection', function(socket) {
    // handling user connection here and pushing users' sockets to
    // users aray.
});

This scrap is basic example of not filled with logic what-you-need. Script should be able to open UDP listener on given port and to listen for users running into it within websockets. Honestly, once you become good in Node.js, you may want to fix both part two + part three with it, what will take UDP part off you as harvester will push output directly to script, that maintains websocket inside it. But it has a drawback of duplicating some logic from other back-end as CRM.

Last ( fourth ) part would be to implement web interface with JavaScript inside, that connects currently logged user to socket server.

I've used similar approach before, and it is working real-time, so we can show our Call-Center employees information about incoming call before even phone actually start to ring. Finally solutions (not counting interface of CRM) closes in two scripts - dedicated CRM API part (where all logic happen) to handle events from Asterisk and Node.js event forwarder.