返回迭代器vs返回Python中的整个列表？

Question

I tested some code for knowing which one is effective, returning iterator and returning whole list.

The program is about reading all lines of .txt file (really big size) and create word counting dictionary (Python3.4).

1.Returning iterator

from collections import defaultdict
import time

def create_word_cnt_dict(line_iter):
    doc_vector = defaultdict(int)
    for line in line_iter:
        for word in line.split():
            doc_vector[word] += 1
    return dict(doc_vector)

def read_doc(doc_file):
    with open(doc_file) as f :
        while True:
            line = f.readline()
            if not line:
                break
            yield line

t0 = time.time()
line_iter = read_doc("./doc1.txt")
doc_vector = create_word_cnt_dict(line_iter)
t1 = time.time()
print(t1-t0)

It takes, 3.765739917755127

2.Returning whole list

from collections import defaultdict
import time

def create_word_cnt_dict(line_list):
    doc_vector = defaultdict(int)
    for line in line_list:
        for word in line.split():
            doc_vector[word] += 1
    return dict(doc_vector)

def read_doc1(doc_file):
    with open(doc_file) as f :
        lines = f.readlines()
        return lines

t0 = time.time()
lines = read_doc1("./doc1.txt")
doc_vector = create_word_cnt_dict(lines)
t1 = time.time()
print(t1-t0)

It takes, 3.6890149116516113

As you can see, returning whole list is much faster.

But in respect of memory usage, Returning iterator is much more effective than returing whole list.

In book Effective Python , it recommends returning iterator for efficient memory usage. But I think that time complexity is more important than space complexity these days because todays computer has enough memory.

Please, give me some advices.

Answer 1

In this case, I think your interpretation of "much faster" is different than mine. . . The timing differences are on the order of a few percent which isn't very much (likely not user noticeable unless your program runs for hours and then the difference is insignificant.)

Couple that with the fact that iterators give you more flexibility. What if you want to stop reading lines when you process a certain one? In that case, the iterator could be factors of 2 or more faster because you've gained the ability to "short circuit".

For the short circuiting reason and memory, I'd prefer the generator function here.

^{Also note that your timings might be biased by the fact that you're reading a file. readlines is probably going to be more efficient because python can read the file in even larger chunks than it normally would which means fewer calls to the OS. Many other applications won't have this sublety...}

Answer 2

Depends.

If we are talking about a relatively small amount of data then time-complexity won't differ either.

Think about huge amount of data and I am not talking about Gbs or TBs , much larger data set that huge companies like Google and Facebook need to handle every day, do you think that space complexity doesn't count as time complexity does?

Space we are not talking about storage memory obviously but for RAM .

So your question is quite broad and it depends on the application, the amount of data that you are going to use and your requirements. For relatively small dataset I don't think that time complexity will be a huge deal neither space complexity.

Answer 3

The performance difference is actually very slight.

In light of that, a good programmer would choose the generator version because it is robust.

If you slurp the whole file you are setting a trap. At some point in the future someone (maybe you) will try to pass in 1GB or 10GB, and they will get screwed over, and run around cursing "WHY??????"