Python statement uses excessive amounts of RAM

Question

This simple statement:

zip(xrange(0, 11614321), xrange(0, 11627964))

...is eating most of my RAM. (>150 MiB!) Why?

Edit: Ah, re-reading the docs, I see zip returns a list, not an iterable. Anything like zip that returns an iterable?

---

The larger picture: I'm iterating over two large arrays of file data, and I'm doing things like iterating (0-end, 0-end), (0-end, 1-end), etc. I'd like to not slice the array, as it would cause excessive allocations of memory. I figured I'd just iterate over the indexes instead, but that doesn't seem to work, as per above. The whole code:

def subsequence_length(data_a, data_b, loc_a, loc_b):
    length = 0
    for i_a, i_b in zip(xrange(loc_a, len(data_a)), xrange(loc_b, len(data_b))):
        if data_a[i_a] == data_b[i_b]:
            length += 1
        else:
            break
    return length

Answer 1

使用itertools中的izip

Answer 2

If for some reason you didn't want to use the itertools module, it would be trivial to write your own iterator that did the same thing, at least if you know you're dealing with exactly two input iterators.

def xzip2(i1, i2):
    i1, i2 = iter(i1), iter(i2)
    while True:
        yield next(i1), next(i2)

Actually, upon further reflection, it is not that hard to make it work with any number of iterators. I am fairly sure itertools.izip must be implemented something like this.

def xzip(*iters):
    iters = [iter(i) for i in iters]
    while True:
        yield tuple([next(i) for i in iters])

(And looking at the documentation , I see it is, except they're using map rather than list comprehensions.)