Handling errors in python with multiple tasks

Question

I have multiple tasks in my python workflow and would like to know what would the best way to handle errors.

class Task1():
    is_ready = False

    def run(self):
        try:
            a = 0/0
            # some more operations
            self._is_ready = True
        except:
            print 'logging errors'


class Task2():
    _is_ready = False

    def run(self):
        try:
            a = 1
            # some more operations
            self._is_ready = True
        except:
            print 'logging errors'

class Workflow():
    def run(self, ):
        self.task1 = Task1()
        self.task2 = Task2()
        self.task1.run()
        if self.task1.is_ready:
            self.task2.run()

w = Workflow()
w.run()         

I basically want to run each tasks sequentially based on the errors of each tasks. ie; if task1 runs fine then process task2..

Can you please let me know will be the above approach will be the right way?

I have totally 10 tasks and thinking adding multiple if loops does not sound like a great way..

Answer 1

If the run() methods could return a boolean success value and each task should only be run if previous succeeded, then it could be done like:

class Workflow():
    def run(self, ):
        task_list = (Task1(), Task2(), Task3(), ...)
        success = all(t.run()  for t in task_list)

Answer 2

There are really two questions here. One is about how to arrange the tasks in a sequence, the other is about how to break the sequence if one task fails.

If you want any kind of scalability, you will need an iterable of tasks, so that you can run a for loop over it. Using nested if s is totally impractical as you yourself noticed. The basic structure will be conceptually something like this:

tasks = [Task1(), Task2(), ...]
for task in tasks:
    task.run()
    if task.failed():
        break

None of the portions of the loop need to appear as written. The loop itself can be replaced with any , all or next . The status check can be an attribute check, a method call or even an implied exception.

You have a number of options for how to decide if a task failed:

1. Use an internal flag as you are currently doing. Make sure that the flag has a consistent name in all the task classes (notice the typo _is_ready in Task2 ). This is a bit of overkill unless you have a use-case that really requires it, since it provides redundant information, and not very elegantly at that.

2. Use a return value in run . This is much nicer because you can write

    for task in tasks: if not task.run(): break 

   Or alternatively (as @MichaelButscher cleverly suggested)

    all(task.run() for task in tasks) 

   In either case, your task should look like this:

    class Task1: def run(self): try: # Some stuff except SomeException: # Log error return False return True 

3. Just let the error propagate from the task implementation:

    class Task1: def run(self): try: # Some stuff except SomeException: # Log error raise 

   I prefer this method to all the others because that's what exceptions are basically for in the first place. In this case, your loop will be even more minimalistic:

    for task in tasks: task.run() 

   Or alternatively, but more obscurely

    any(task.run() for task in tasks) 

   Or even

    from collections import deque deque(task.run() for task in tasks, maxlen=0) 

   The second two options are really there only for reference purposes. If you go with exceptions, just write the basic for loop: it's plenty elegant enough and by far the least arcane.

Finally, I would recommend another fundamental change. If your tasks are truly arbitrary in nature, then you should consider allowing any callable taking no arguments to be a task. There is no particular need to restrict yourself to classes having a run method. If you need to have a task class, you can rename the method you call run to __call__ , and all your instances will be callable with the () operator. The code would look conceptually like this then:

class CallableClass:
    def __call__(self):
        try:
            # Do something
        except:
            # Log error
            raise

def callable_function():
    try:
        # Do something
    except:
        # Log error
        raise

for task in tasks:
    task()