Improve speed for list concatenation?

Question

I have a list called L inside a loop that must iterate though millions of lines. The salient features are:

for line in lines:
    L = ['a', 'list', 'with', 'lots', 'of', 'items']
    L[3] = 'prefix_text_to_item3' + L[3]
    Do more stuff with L...

Is there a better approach to adding text to a list item that would speed up my code. Can .join be used? Thanks.

Answer 1

In a performance oriented code, it is not a good idea to add 2 strings together, it is preferable to use a "".join(_items2join_) instead. (I found some benchmarks there : http://www.skymind.com/~ocrow/python_string/ )

Answer 2

Since accessing an element in a python list is O(1), and appending a list to another is O(1) (which is probably the time complexity of concatenating strings in python), The code you have provided is running as fast as it can as far as I can tell. :) You probably can't afford to do this, but when I need speed I go to C++ or some other compiled language when I need to process that much information. Things run much quicker. For time complexity of list operations in python, you may consult this web site: http://wiki.python.org/moin/TimeComplexity and here: What is the runtime complexity of python list functions?

Answer 3

Don't actually create list objects.

Use generator functions and generator expressions.

def appender( some_list, some_text ):
    for item in some_list:
        yield item + some_text

This appender function does not actually create a new list. It avoids some of the memory management overheads associated with creating a new list.

Answer 4

There may be a better approach depending on what you are doing with list L.

For instance, if you are printing it, something like this may be faster.

print "{0} {1} {2} {3}{4} {5}".format(L[0], L[1], L[2], 'prefix_text_to_item3', L[3], L[4])

What happens to L later in the program?