Python结构和执行时间

Question

I am working on an application and have a choice between the following two basic structures. I ran a small test to determine the difference in execution time between the two. The second option is 758 times faster than the first option. What can I do to retain the structure of the first option with the execution speed of the second? It will be very messy to code option two (my example is vastly scaled down). The application is not computationally intensive it just involves a lot of data and small computations. I don't want to get into PyPy or Cython.

What is the causing the first option to run slow relative to the second? Is it the creation of the list c[] ten thousand times? I thought the Python interpreter was smart enough to anticipate. Or is it the call of def b()?

Option 1:

#! /usr/bin/env python

def a(b):

    c=[10 for i in range(10000)]
    return c[b]

def b():
    i=0
    while i < 10000:
        d = a(i)**a(i)^a(i)**a(i)
        i += 1

b()

Option 2:

#! /usr/bin/env python

a=[10 for i in range(10000)]

i=0
while i < 10000:
    d = a[i]**a[i]^a[i]**a[i]
    i += 1

Answer 1

This will give you the same result, and the structure is similar to option 1. This will be far more efficient than option 1.

a = [10 for i in range(10000)]

def b():
    for c in a:
        d = c ** c ^ c ** c

b()

Option 1 is slower than option 2 because you are repeatedly calling a which tends to slow the execution and is not very efficient.

Answer 2

You give the answer in your question. In option one you create the list c for each and every call to a . Python does not have the concept of static variables in function and the interpreter/JIT will most likely not optimize that away. You have several options to move list c out of the scope of a to prevent a recreation.

Option 1 - Global variable

c=[10 for i in range(10000)]

def a(b):
    return c[b]

def b():
    i=0
    while i < 10000:
        d = a(i)**a(i)^a(i)**a(i)
        i += 1

b()

Global variables are generally not very nice, though. At this point the point of function a is also very limited and you might as well remove it. Never expect a JIT to do optimizations as good as good as an AOT compiler who would very likely inline a .

Option 2 - A function closure

def createA():
    c=[10 for i in range(10000)]
    def a(b):
        return c[b]
    return a

a = createA()

def b():
    i=0
    while i < 10000:
        d = a(i)**a(i)^a(i)**a(i)
        i += 1

b()

This is very JavaScript-esque and would much nicer if Python had a more extensive support for anonymous functions. This doesn't look very nice, either, since now you pollute the global scope with another function.

Option 3 - Functors

class A(object):
    def __init__(self):
        self._c = [10 for i in range(10000)]

    def __call__(self, b):
        return self._c[b]

a = A()

def b():
    i=0
    while i < 10000:
        d = a(i)**a(i)^a(i)**a(i)
        i += 1

b()

Note that you are adding the class A to the global namespace.

Option 4 - Set the "look up list" on the function

def a(b):
    return a._c[b]

a._c=[10 for i in range(10000)]

def b():
    i=0
    while i < 10000:
        d = a(i)**a(i)^a(i)**a(i)
        i += 1

b()

Here you are retaining structure, not polluting the global namespace and not recreating the list on each call. This is basically as close as you get to static function variables in Python. If a will be very simple you still want to consider inlining the functionality for performance reasons.

Answer 3

As you mentioned, recreating list 'c' for 10k times is the culprit. From system point of running this code via ltrace shows below

$ ltrace -fc  /usr/bin/python ./opt1.py 2>&1 |head -5
% time     seconds  usecs/call     calls      function
------ ----------- ----------- --------- --------------------
 59.08    7.751628          71    108965 realloc
  8.33    1.093207          69     15651 memset
  7.35    0.964546          69     13825 memcpy

$ ltrace -fc  /usr/bin/python ./opt2.py 2>&1 |head -5
% time     seconds  usecs/call     calls      function
------ ----------- ----------- --------- --------------------
 28.37    0.973063          70     13797 memcpy
 15.58    0.534149          70      7615 memset
 14.20    0.487080          69      6975 strlen

python is calling realloc every time you extend list and thus causing slowness observed. I ran test with version 2.7.9 of python. Profiling within python would be the best thing to check further, but i just used quick/dirty way to look

PS: my test was reduced to 1k iterations only