将 append 串入 python 的正确方法

Question

I've read this reply which explains that CPython has an optimization to do an in-place append without copy when appending to a string using a = a + b or a += b . I've also read this PEP8 recommendation:

Code should be written in a way that does not disadvantage other implementations of Python (PyPy, Jython, IronPython, Cython, Psyco, and such). For example, do not rely on CPython's efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b. This optimization is fragile even in CPython (it only works for some types) and isn't present at all in implementations that don't use refcounting. In performance sensitive parts of the library, the ''.join() form should be used instead. This will ensure that concatenation occurs in linear time across various implementations.

So if I understand correctly, instead of doing a += b + c in order to trigger this CPython optimization which does the replacement in-place, the proper way is to call a = ''.join([a, b, c]) ?

But then why is this form with join significantly slower than the form in += in this example (In loop1 I'm using a = a + b + c on purpose in order to not trigger the CPython optimization)?

import os
import time

if __name__ == "__main__":
    start_time = time.time()
    print("begin: %s " % (start_time))
    s = ""
    for i in range(100000):
        s = s + str(i) + '3'
    time1 = time.time()
    print("end loop1: %s " % (time1 - start_time))

    s2 = ""
    for i in range(100000):
        s2 += str(i) + '3'

    time2 = time.time()
    print("end loop2: %s " % (time2 - time1))

    s3 = ""
    for i in range(100000):
        s3 = ''.join([s3, str(i), '3'])

    time3 = time.time()
    print("end loop3: %s " % (time3 - time2))

The results show join is significantly slower in this case:

~/testdir$ python --version
Python 3.10.6
~/testdir$ python concatenate.py 
begin: 1675268345.0761461 
end loop1: 3.9019 
end loop2: 0.0260 
end loop3: 0.9289 

Is my version with join wrong?

Answer 1

In "loop3" you bypass a lot of the gain of join() by continuously calling it in an unneeded way. It would be better to build up the full list of characters then join() once.

Check out:

import time

iterations = 100_000

##----------------
s = ""
start_time = time.time()
for i in range(iterations):
    s = s + "." + '3'
end_time = time.time()
print("end loop1: %s " % (end_time - start_time))
##----------------

##----------------
s = ""
start_time = time.time()
for i in range(iterations):
    s += "." + '3'
end_time = time.time()
print("end loop2: %s " % (end_time - start_time))
##----------------

##----------------
s = ""
start_time = time.time()
for i in range(iterations):
    s = ''.join([s, ".", '3'])
end_time = time.time()
print("end loop3: %s " % (end_time - start_time))
##----------------

##----------------
s = []
start_time = time.time()
for i in range(iterations):
    s.append(".")
    s.append("3")
s = "".join(s)
end_time = time.time()
print("end loop4: %s " % (end_time - start_time))
##----------------

##----------------
s = []
start_time = time.time()
for i in range(iterations):
    s.extend((".", "3"))
s = "".join(s)
end_time = time.time()
print("end loop5: %s " % (end_time - start_time))
##----------------

Just to be clear, you can run this with:

iterations = 10_000_000

If you like, just be sure to remove "loop1" and "loop3" as they get dramatically slower after about 300k.

When I run this with 10 million iterations I see:

end loop2: 16.977502584457397 
end loop4: 1.6301295757293701 
end loop5: 1.0435805320739746

So, clearly there is a way to use join() that is fast:-)