使用Python API的Ansible Playbook重试逻辑

Question

使用Ansible 2 Python API，我可以运行剧本并使用自定义的回调处理程序处理结果（由于这个问题 ）。 一切正常，但现在我想为PlaybookExecutor实现一个简单的重试循环。

我所有的回调处理程序所做的就是将所有失败的任务填充到数组中，如果我看到该数组不为空，则将其视为失败，然后重试。

我还有另一个使用此脚本启动剧本的python模块。 对run_playbook的调用嵌套在try / except块中，我希望冒出一个异常，以便我能够正确处理故障。

我想给我的剧本3次运行尝试，如果都失败了，那就抛出一个异常。

这是我的代码：

#! /usr/bin/python
# -*- coding: utf-8 -*-

from __future__ import print_function
import logging
import os
from collections import namedtuple
from ansible.parsing.dataloader import DataLoader
from ansible.vars.manager import VariableManager
from ansible.inventory.manager import InventoryManager
from ansible.executor.playbook_executor import PlaybookExecutor
from ansible.plugins.callback import CallbackBase

class ResultsCallback(CallbackBase):
    """ A callback plugin used for performing an action as results come in """
    def __init__(self):
        super(ResultsCallback, self).__init__()
        # Store all failed results
        self.failed = []

    def v2_runner_on_failed(self, result, ignore_errors=False):
        if ignore_errors:
            self._display.display("...ignoring", color=C.COLOR_SKIP)
        host = result._host
        self.failed.append(result.task_name)


def create_inventory_file(hostnames):
    inv_file = 'ansible_hosts.{0}'.format(os.getppid())
    logging.print('\nCreating Ansible host file: {0}/{1}'.format(os.path.join(os.path.expanduser('~')), inv_file))
    with open(os.path.join(os.path.expanduser('~'), inv_file), 'w') as host_file:
        # If ec2, stuff into an '[ec2]' group.
        # Otherwise don't use a group header
        if 'ec2' in hostnames[0]:
            host_file.write('[ec2]\n')
        for host in hostnames:
            host_file.write('{0}\n'.format(host))

    return os.path.join(os.path.expanduser('~'), inv_file)


def run_playbook(hostnames, playbook, playbook_arguments, host_file=False):
    # If user passes in the optional arg host_file, then just use that one.
    if not host_file:                                                                                                                                                                                          
        host_file = create_inventory_file(hostnames)
    if not os.path.isfile(host_file):
        logging.critical('Host file does not exist. Make sure absolute path is correct.\nInventory: {0}'.format(host_file))
        raise RuntimeError('Host file does not exist')

    loader = DataLoader()
    inventory = InventoryManager(loader=loader, sources=host_file)
    variable_manager = VariableManager(loader=loader, inventory=inventory)

    # Add extra variables to use in playbook like so:
    # variable_manager.extra_vars = {'name': 'value'}
    if playbook_arguments:
        variable_manager.extra_vars = playbook_arguments

    Options = namedtuple('Options', ['listtags', 'listtasks', 'listhosts', 'syntax', 'connection','module_path', 'forks', 'remote_user', 'become', 'become_method', 'become_user', 'verbosity', 'check', 'diff', 'ask_sudo_pass'])

    if 'superuser' in playbook_arguments:
        remote_user = playbook_arguments['superuser']
    else:
        remote_user = 'ec2-user'

    options = Options(listtags=None, listtasks=None, listhosts=None, syntax=None, connection='smart', module_path=None, forks=100, remote_user=remote_user,  become=None, become_method='sudo', become_user='root', verbosity=None, check=False, diff=False, ask_sudo_pass=None)

    pbex = PlaybookExecutor(playbooks=[playbook], inventory=inventory, variable_manager=variable_manager, loader=loader, options=options, passwords={})

    callback = ResultsCallback()
    pbex._tqm._stdout_callback = callback

    logging.print('Provisioning cluster with Ansible...')
    attempts = 3
    for i in range(attempts):
        try:
            pbex.run()
            failed = callback.failed
            if failed:
                logging.critical('Playbook failed!')
                raise RuntimeError('{0} tasks failed'.format(len(failed)))
            break
        except:
            if i < attempts - 1:
                logging.critical('Attempting to re-try playbook')
                continue
            else:
                raise

    logging.print('\nRemoving Ansible Inventory file {0}'.format(host_file))
    try:
        os.remove(host_file)
    except OSError:
        pass

但是，当我使用肯定会失败的剧本测试上述代码时，它将失败，并产生以下回溯：

Creating Ansible host file: /home/someuser/ansible_hosts.18600
Provisioning cluster with Ansible...
Playbook failed!
Attempting to re-try playbook
Exception during setup; tearing down all created instances
Traceback (most recent call last):
  File "./manage_aws.py", line 486, in cmd_ec2_create
    manage_ansible.run_playbook(hostnames, playbook, playbook_arguments)
  File "/home/someuser/manage_ansible.py", line 88, in run_playbook
    break
  File "/usr/local/lib/python2.7/dist-packages/ansible/executor/playbook_executor.py", line 159, in run
    result = self._tqm.run(play=play)
  File "/usr/local/lib/python2.7/dist-packages/ansible/executor/task_queue_manager.py", line 296, in run
    strategy.cleanup()
  File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/strategy/__init__.py", line 223, in cleanup
    self._final_q.put(_sentinel)
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 100, in put
    assert not self._closed
AssertionError

您会注意到，该异常已正确地捕获在调用脚本manage_aws.py中（“安装过程中的异常；拆除所有创建的实例”），我们将拆除实例。 太好了，但是我想在决定这样做之前重新尝试该剧本。

我不是Python的高手，所以如果有人有任何建议或完成类似的工作，那么我将非常感谢您的建议。

提前致谢！

Answer 1

我找到了一个解决方案，虽然它不像我希望的那样优雅。

我遇到的问题似乎与在同一PlaybokExecutor对象上重新运行而没有正确清理生成的线程有关。

我要修复的问题是，当我发现第一个失败时，只是初始化了一个新的PlaybookExecutor对象。 当前的实现只允许重试一次，这很好，但是如果有必要，我很可能会对其进行修改以执行更多操作。

这是我改编的重试逻辑：

    pbex = PlaybookExecutor(playbooks=[playbook], inventory=inventory, variable_manager=variable_manager, loader=loader, options=options, passwords={})

    callback = ResultsCallback()
    pbex._tqm._stdout_callback = callback

    logging.print('Provisioning cluster with Ansible...')
    pbex.run()
    failed = callback.failed
    if failed:
        logging.critical('Playbook failed! Attempting retry...')
        pbex_retry = PlaybookExecutor(playbooks=[playbook], inventory=inventory, variable_manager=variable_manager, loader=loader, options=options, passwords={})
        callback_retry = ResultsCallback()
        pbex_retry._tqm._stdout_callback = callback_retry
        pbex_retry.run()
        failed_retry = callback_retry.failed
        if failed_retry:
            logging.critical('Playbook failed again! Failed on task:\n{0}'.format(failed_retry[0]))
            remove_inventory_file(host_file)
            raise RuntimeError('Playbook failed to successfully configure the cluster.')

    remove_inventory_file(host_file)

超级简单的解决方案，可惜我最初的尝试未能按预期进行。 也许我会重新访问它，并在失败时尝试正确清理执行程序。