best way to store code for later execution (python)

Question

I have some tasks stored in db for later execution. For example i can fix task of sending email. And by cron exec task (send it). I search for best way to store code in db for later execution. For ex store it in raw string of python code and than do eval, but also i must store relative imports here..

for example for send email i must fix string like this:

s = "from django.core.mail import send_mail\\n send_mail('subj', 'body', 'email@box.ru',['email1@box.ru'], fail_silently=False)"

and later eval.. any ideas to do it best way or mb better pattern for this kind of task?

Answer 1

What you're doing is a bad idea mainly because you allow for way too much variability in what code will be executed. A code string can do anything , and I'm guessing there are only a few kinds of tasks you want to store for later execution.

So, figure out what the variables in those tasks are (variables in a non-programming sense: things that vary), and only store those variables, perhaps as a tuple of function arguments and a dictionary of keyword arguments to be applied to a known function.

To be even more fancy, you can have some kind of container object with a bunch of functions on it, and store the name of the function to call along with its arguments. That container could be something as simple as a module into which you import functions like Django's send_mail as in your example.

Then you can store your example call like this:

func = 'send_mail'
args = ('subj', 'body', 'email@box.ru', ['email1@box.ru'])
kwargs = {'fail_silently': False}

my_call = cPickle.dumps((func, args, kwargs)) 

And use it like this:

func, args, kwargs = cPickle.loads(my_call)

getattr(my_module, func)(*args, **kwargs)

Answer 2

I wouldn't use this solution at all. I would create a different handler for each task (sending a mail, deleting a file, etc). Storing code in this manner is hackish.

EDIT

An example would be creating your own format for handlers. For example each line one handler in this format:

handlername;arg1;arg2;arg3;arg4

Next you use python to read out the lines and parse them. For example this would be a stored line:

sendmail;nightcracker@nclabs.org;subject;body

Which would be parsed like this:

for line in database:
    handler, *args = line.split(";")
    if handler == "sendmail":
        recipient, subject, body, = args[:3]
        # do stuff
    elif handler == "delfile":
        #etc

Answer 3

Use celery for this. That's the best approach.

http://celeryproject.org/

Answer 4

I'd store logical commands, and exec them with something like

def run_command(cmd):
    fields = map(unescape, cmd.split(";"))
    handlers[fields[0]](fields[1:])

...

@handler("mail")
def mail_handler(address, template):
    import whatever
    ...
    send_mail(address, get_template(template) % user_info, ...)

this way you can have both the flexibility to add handlers without having to touching any code in the dispatcher and yet you're not writing the code details in the database that would make harder doing inspections/stats or just hot fixing jobs that didn't start yet.

Answer 5

To directly answer your question, eval is really only for evaluating code that will produce a result. For example:

>>> eval('1 + 1')
2

However if you simply want to execute code, possibly several lines of code, you want exec(), which by default executes inside the caller's namespace:

>>> exec("x = 5 + 5")
>>> print x
10

Note that only trusted code should be passed to either exec or eval. See also execfile to execute a file.

Having said all that, I agree with other posters that you should find a way to problematically do what you want to do instead of storing arbitrary code. You could, for example, do something like this:

def myMailCommand(...):
    ...

def myOtherCommand(...):
    ...

available_commands = {'mail': myMailCommand,
                      'other': myOtherCommand}

to_execute = [('mail', (arg1, arg2, arg3)),
              ('other', (arg1, arg2))]

for cmd, args in to_execute:
    available_commands[cmd](*args)

In the above pseudo-code, I defined two methods. Then I have a dictionary mapping actions to commands. Then I go through a data structure of actions and arguments, and call the appropriate argument accordingly. You get the idea.