使用Python自動進行流程監控/管理

Question

是的，所以我有一個不斷運行的python進程，甚至在Supervisor上也可以。 實現以下監視的最佳方法是什么？

• 發送警報，如果進程崩潰，則重新啟動。 我希望每次進程崩潰時自動接收信號，然后自動重新啟動它。
• 發送警報，如果流程過時，即1分鍾未處理任何內容，則重新啟動。
• 按需重啟

我想通過Python實現以上所有功能。 我知道Supervisord將完成大部分操作，但是我想看看它是否可以通過Python本身完成。

Answer 1

我認為您正在尋找的是Supervisor Events。 http://supervisord.org/events.html

還要看一下Superlance，它是一套插件實用程序，用於監視和控制在主管下運行的進程。 [ https://superlance.readthedocs.org/en/latest/]

您可以配置崩潰電子郵件，崩潰短信，內存消耗警報，HTTP掛鈎等內容。

Answer 2

好吧，如果您想要一個本地解決方案，這就是我能想到的。

保持進程狀態在Redis中處於實際狀態和預期狀態。 您可以通過使Web界面檢查實際狀態並更改預期狀態來以所需方式對其進行監視。

在crontab中運行python腳本以檢查狀態，並在需要時采取適當的措施。 在這里，我每3秒鍾檢查一次，並使用SES通過電子郵件提醒管理員。

免責聲明：該代碼尚未運行或測試。 我現在才寫，容易出錯。

打開crontab文件：

$crontab -e

在其末尾添加此行，以使run_process.sh每分鍾運行一次。

#Runs this process every 1 minute.
*/1 * * * * bash ~/path/to/run_monitor.sh

run_moniter.sh運行python腳本。 它每3秒在for循環中運行一次。

這樣做是因為crontab給出了1分鍾的最小時間間隔。 我們想每3秒檢查一次該過程20次（3秒* 20 = 1分鍾）。 因此它將運行一分鍾，然后crontab再次運行它。

run_monitor.sh

for count in {0..20}
do
    cd '/path/to/check_status'
    /usr/local/bin/python check_status.py "myprocessname" "python startcommand.py"
    sleep 3 #check every 3 seconds.
done

我在這里假設：

*狀態0 =停止或停止（預期與實際）

*狀態-1 =重新啟動

*狀態1 =運行或正在運行

您可以根據需要添加更多狀態，過時的過程也可以是狀態。

我使用過進程名來殺死或啟動或檢查進程，您可以輕松地對其進行修改以讀取特定的PID文件。

check_status.py

import sys
import redis
import subprocess

import sys
import boto.ses


def send_mail(recipients, message_subject, message_body):
    """
    uses AWS SES to send mail.
    """
    SENDER_MAIL = 'xxx@yyy.com'
    AWS_KEY = 'xxxxxxxxxxxxxxxxxxx'
    AWS_SECRET = 'xxxxxxxxxxxxxxxxxxx'
    AWS_REGION = 'xx-xxxx-x'

    mail_conn = boto.ses.connect_to_region(AWS_REGION, 
                                           aws_access_key_id=AWS_KEY, 
                                           aws_secret_access_key=AWS_SECRET
                                           )

    mail_conn.send_email(SENDER_MAIL, message_subject, message_body, recipient, format='html')
    return True

class Shell(object):
    '''
    Convinient Wrapper over Subprocess.
    '''
    def __init__(self, command, raise_on_error=True):
        self.command = command
        self.output = None
        self.error = None
        self.return_code

    def run(self):
        try:
            process = subprocess.Popen(self.command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
            self.return_code = process.wait()
            self.output, self.error = process.communicate()
            if self.return_code and self.raise_on_error:
                print self.error
                raise Exception("Error while executing %s::%s"%(self.command, self.error))    
        except subprocess.CalledProcessError:
            print self.error
            raise Exception("Error while executing %s::%s"%(self.command, self.error))


redis_client = redis.Redis('xxxredis_hostxxx')

def get_state(process_name, state_type): #state_type will be expected or actual.
    state = redis.get('{process_name}_{state_type}_state'.format(process_name=process_name, state_type=state_type)) #value could be 0 or 1
    return state

def set_state(process_name, state_type, state): #state_type will be expected or actual.
    state = redis.set('{process_name}_{state_type}_state'.format(process_name=process_name, state_type=state_type), state)
    return state

def get_stale_state(process_name):
    state = redis.get('{process_name}_stale_state'.format(process_name=process_name)) #value could be 0 or 1
    return state

def check_running_status(process_name):
    command = "ps -ef|grep {process_name}|wc -l".format(process_name=process_name)
    shell = Shell(command = command)
    shell.run()
    if shell.output=='0':
        return False
    return True

def start_process(start_command): #pass start_command with a '&' so the process starts in the background.
    shell = Shell(command = command)
    shell.run()

def stop_process(process_name):
    command = "ps -ef| grep {process_name}| awk '{print $2}'".format(process_name=process_name)
    shell = Shell(command = command, raise_on_error=False)
    shell.run()
    if not shell.output:
        return
    process_ids = shell.output.strip().split()
    for process_id in process_ids:
        command = 'kill {process_id}'.format(process_id=process_id)
        shell = Shell(command=command, raise_on_error=False)
        shel.run()


def check_process(process_name, start_command):
    expected_state = get_state(process_name, 'expected')
    if expected_state == 0: #stop
        stop_process(process_name)
        set_state(process_name, 'actual', 0)

    else if expected_state == -1: #restart
        stop_process(process_name)
        set_state(process_name, 'actual', 0)
        start_process(start_command)
        set_state(process_name, 'actual', 1)
        set_state(process_name, 'expected', 1) #set expected back to 1 so we dont keep on restarting.

    elif expected_state == 1:
        running = check_running_status(process_name)
        if not running:
            set_state(process_name, 'actual', 0)
            send_mail(reciepients=["abc@admin.com", "xyz@admin.com"], message_subject="Alert", message_body="Your process is Down. Trying to restart")
            start_process(start_command)
            running = check_running_status(process_name)
            if running:
                send_mail(reciepients=["abc@admin.com", "xyz@admin.com"], message_subject="Alert", message_body="Your process is was restarted.")
                set_state(process_name, 'actual', 1)
            else:
                send_mail(reciepients=["abc@admin.com", "xyz@admin.com"], message_subject="Alert", message_body="Your process is could not be restarted.")


if __name__ == '__main__':
    args = sys.argv[1:]
    process_name = args[0]
    start_command = args[1]
    check_process(process_name, start_command)