improving speed on strings with math operators in python

Question

I am working with a set of strings that could contain math operators, either '+' or '-' for now. Initially i used eval which was pretty straightforward. I am trying to move away from eval due to its known issues but mostly am trying to improve the speed. Here is a typical set of string that i would work on :

DAC = ('12','0x1E','34','12+20','2+0x1F')

Taking each value and applying the eval will give the results i need. To move away from this I tried a few methods, they work fine until i hit into a math operator. I did read about AST module and am gonna work on that but would like to get some feedback if its the right track to go in terms of improving the speed or any other suggestions the community can give me..

Thanks

Answer 1

What is wrong with literal_eval ? That's probably your best bet short of writing or finding a lex parser, which is certainly overkill for this situation.

>>> DAC = ('12','0x1E','34','12+20','2+0x1F')
>>> from ast import literal_eval

You could use a list comprehension

>>> [literal_eval(i) for i in DAC]
[12, 30, 34, 32, 33]

Or map

>>> list(map(literal_eval, DAC))
[12, 30, 34, 32, 33]

Answer 2

As I measured, in Python 3 where it's able to use operators, ast.literal_eval is a bit slower than plain eval . So the fastest approach might be:

def myeval(s):
    try: return int(s, 0)
    except ValueError: return eval(s)

Having placed this and your definition of DAC in x.py, I then measure:

aleax-macbookair4:~ aleax$ python3 -mtimeit -s'import x' '[eval(a) for a in x.DAC]'
10000 loops, best of 3: 34 usec per loop
aleax-macbookair4:~ aleax$ python3 -mtimeit -s'import x' '[x.myeval(a) for a in x.DAC]'
10000 loops, best of 3: 22.9 usec per loop

so, a modest speedup.

Actually, for once, a "LBYL" approach may be a little bit better than the usually preferable "EAFP" one: adding to x.py the following

def alteval(s):
    if '+' in s or '-' in s: return eval(s)
    else: return int(s, 0)

I then measure:

aleax-macbookair4:~ aleax$ python3 -mtimeit -s'import x' '[x.alteval(a) for a in x.DAC]'
100000 loops, best of 3: 18.1 usec per loop

for another, though smaller, increase in performance. (I've also tried using re to find out if s has a plus or minus, but that appears to be slower).