Python性能比较：将Iterator转换为List并按索引添加元素（预分配后）与在迭代时追加

Question

Long story short. What is better:

Given an iterator: eg. After Reading a CSV or getting query results from DB. What would give better performance and why?

First Approach: Iterate using the iterator and append to required lists. Something like:

element1_list=[]
element2_list=[]
for row in rows:
    element1_list.append(row[element1_index])
    element2_list.append(row[element2_index])

Second Approach: Convert the iterator to a list and access the elements after preallocation

row_list=list(rows)
length=len(row_list)
element1_list=[None]*length
element2_list=[None]*length
for i in range(0,length):
    element1_list[i]=row_list[i][element1_index]
    element2_list[i]=row_list[i][element2_index]

Preallocation has it's own benefits. But conversion to a list, may itself be an iteration itself. So what approach to choose and why? Would be interesting to know what happens under the hood?

EDIT : Again emphasizing, would like to know about the fundamental differences in these approaches. NOT merely using timeit and doing empirical analysis, which i would like to do to back up theory and not the other way around

Some of the performance criteria maybe:

1. Speed and CPU time
2. Memory

Answer 1

Since everyone recommends timeit , so here are the results and the accompanying code.

The code i used for testing is as follows:

import timeit
import matplotlib.pyplot as plt
import csv

index_element_1=0
index_element_2=2

def create_csv(num_records):
    '''Creates a test CSV'''
    a=open('test.csv','wb')
    a.write("10,20,30,40,50\n"*num_records)
    a.close()

def read_csv(filename):
    '''Returns iterator'''
    cr = csv.reader(open(filename,"rb"))
    return cr

def convert_to_list_method():
    global csv_iterator
    csv_list=list(csv_iterator)
    length_list=len(csv_list)
    x=[None]*length_list
    y=[None]*length_list
    for i in range(0,length_list):
        x[i]=csv_list[index_element_1]
        y[i]=csv_list[index_element_2]
    return [x,y]

def iterate_and_append_method():
    global csv_iterator
    x=[]
    y=[]
    for row in csv_iterator:
        x.append(row[index_element_1])
        y.append(row[index_element_2])
    return [x,y]


CSV_SIZE=range(10000,1010000,10000)
time_convert_to_list=[0]*len(CSV_SIZE)
time_iterate=[0]*len(CSV_SIZE)
count=0

for csv_size in CSV_SIZE:
    create_csv(csv_size)
    global csv_iterator
    csv_iterator=read_csv('test.csv')
    time_convert_to_list[count]=timeit.timeit("convert_to_list_method()", setup="from __main__ import *",number=1)
    csv_iterator=read_csv('test.csv')
    time_iterate[count]=timeit.timeit("iterate_and_append_method()", setup="from __main__ import *",number=1)
    count=count+1

plt.xlabel('CSV Size')
plt.ylabel('Time (s)')
plt.plot(CSV_SIZE,time_convert_to_list,label='Convert to List')
plt.plot(CSV_SIZE,time_iterate,label='Iterate')
plt.legend()
plt.show()

The results don't vary much. I think all the comments were right. It does not really make a lot of difference.

NB: I used only 1 iteration of each function in timeit since o/w the iterator would have to be recreated, since it got consumed by previous iteration!