启动和停止集群中的进程

Question

I'm writing software that runs a bunch of different programs (via twisted's twistd ); that is N daemons of various kinds must be started across multiple machines. If I did this manually, I would be running commands like twistd foo_worker , twistd bar_worker and so on on the machines involved.

Basically there will be a list of machines, and the daemon(s) I need them to run. Additionally, I need to shut them all down when the need arises.

If I were to program this from scratch, I would write a "spawner" daemon that would run permanently on each machine in the cluster with the following features accessible through the network for an authenticated administrator client:

• Start a process with a given command line. Return a handle to manage it.
• Kill a process given a handle.
• Optionally, query stuff like cpu time given a handle.

It would be fairly trivial to program the above, but I cannot imagine this is a new problem. Surely there are existing solutions to doing exactly this? I do however lack experience with server administration, and don't even know what the related terms are.

What existing ways are there to do this on a linux cluster, and what are some of the important terms involved? Python specific solutions are welcome, but not necessary.

Another way to put it: Given a bunch of machines in a lan, how do I programmatically work with them as a cluster?

Answer 1

通常的工具是批处理队列系统，例如SLURM，SGE，Torque / Moab，LSF等。

Answer 2

The most familiar and universal way is just to use ssh . To automate you could use fabric .

To start foo_worker on all hosts:

$ fab all_hosts start:foo_worker

To stop bar_worker on a particular list of hosts:

$ fab -H host1,host2 stop:bar_worker

Here's an example fabfile.py :

from fabric.api import env, run, hide # pip install fabric

def all_hosts():
    env.hosts = ['host1', 'host2', 'host3']

def start(daemon):
    run("twistd --pid %s.pid %s" % (daemon, daemon))

def stop(daemon):
    run("kill %s" % getpid(daemon))

def getpid(daemon):
    with hide('stdout'):
        return run("cat %s.pid" % daemon)

def ps(daemon):
    """Get process info for the `daemon`."""
    run("ps --pid %s" % getpid(daemon))

There are a number of ways to configure host lists in fabric, with scopes varying from global to per-task, and it's possible mix and match as needed. .

To streamline the process management on a particular host you could write initd scripts for the daemons (and run service daemon_name start/stop/restart ) or use supervisord (and run supervisorctl eg, supervisorctl stop all ). To control "what installed where" and to push configuration in a centralized manner something like puppet could be used.

Answer 3

Circus :

Documentation : http://docs.circus.io/en/0.5/index.html

Code: http://pypi.python.org/pypi/circus/0.5

Summary from the documentation :

Circus is a process & socket manager. It can be used to monitor and control processes and sockets.

Circus can be driven via a command-line interface or programmatically trough its python API.

It shares some of the goals of Supervisord, BluePill and Daemontools. If you are curious about what Circus brings compared to other projects, read Why should I use Circus instead of X ?.

Circus is designed using ZeroMQ http://www.zeromq.org/ . See Design for more details.