为什么python中字符串连接的顺序会大大影响速度？

Question

I just found this issue through debugging my code. I had a list of messages as strings that I was trying to concatenate together, and I wanted to add a newline to the end of every message.

Approach 1:

total_str = ""
for m in messages:
    total_str = total_str + m + "\n"

This was extremely slow - after around the 100,000th message, adding each message took about 2-3s, and around the 300,000th message, this process basically stopped.

Approach 2:

total_str = ""
for m in messages:
    tmp = m + "\n"
    total_str = total_str + tmp

This approach finished concatenating all 1.6 million messages in less than a second.

What I'm wondering is why is the second approach so much faster than the first?

Answer 1

a + b + c isn't a single operation that joins a , b , and c into a single string. It is two operations, t = a + b and t + c , which means copying the contents of a twice ; once to copy a into t , and again when t gets copied into the result of t + c . Since, in your example, a is the string that keeps getting longer, you are at best doubling the amount of data being copied at each step.

The best approach is to avoid all the temporary str object created by + , and use join :

total_str = "\n".join(messages)

join operates with each string directly, without the need to iteratively append them to an initial empty string one at a time. join figures out, by scanning messages , how long the resulting string needs to be, allocates enough memory for it, then sequentially copies the data from each element of messages into place one at a time.

Answer 2

Well, since a = a + b + c is executed as a = (a + b) + c , one can see that the order of computation is the following:

• tmp_1 = a + b . This has to copy the huge string a because strings are immutable.
• a = tmp_1 + c . This has to copy the (even more) huge string tmp_1 because strings are immutable.

So, there are two huge copies involved, while in the second version, a = a + tmp (like in your second example), only one such copy is needed. The latter approach will obviously be faster.

Answer 3

Python's strings are immutable and contiguous. The former means they can't be modified, and the latter means they're stored in one place in memory. This is unlike eg a rope data structure , where appending data is a cheap operation that need only form a new node for the end. It means that the concatenation operation must copy both input strings each time, and with something like total_str = total_str + m + "\\n" , since + is left associative , copies all of total_str twice. The usual solution is keeping all the small strings until the whole set is completed, and using str.join to perform the concatenations in one pass. This would only copy each component string once, instead of a geometric (proportional to square) number of times. Another option, to build a buffer as you go along, is to use io.StringIO . That will give you a file-like object, a bit like a StringBuilder in some other languages, from which you can extract the final string. We also have operations like writelines that can accept iterables, so the join may not be needed at all.

My guess as for why the second implementation managed to be so much faster (not just about twice as fast), is that there are optimizations in place that can sometimes permit CPython not to perform the copy of the left operand at all. PyUnicode_Append appears to have precisely such an optimization, based on unicode_modifiable , wherein it can mutate an object if the reference count is precisely 1, the string has never been hashed, and a few other conditions. This would typically apply to a local variable where you use += , and presumably the compiler managed to generate such behaviour when there wasn't a second operator in the same assignment.