Python 2.7: “AttributeError: 'callable-iterator' object has no attribute 'group' ” (regex finditer within generator expression)

Question

I'm writing a program that involves the following generator expression (as you might guess, 'sep2' is a compiled regular expression):

(chunk.group(1).strip(punctuation) for chunk in sep2.finditer(textString))

This works fine. However, it occurred to me that the program would be more scalable if instead of working with a big string, it worked through a file line by line. So I tried the following:

(chunk.group(1).strip(punctuation) for chunk in (sep2.finditer(line) for line in file))

Unfortunately, this throws the following error message:

AttributeError: 'callable-iterator' object has no attribute 'group'

Now, I could just give up on the idea of working through the file line by line - to be honest, it's not a big deal. But I'm curious to know what's going wrong here. I've searched stackoverflow (and the Python website, and the internet generally) for the error message above but haven't found anything that seems relevant. I think that what I need to know is how to get the match objects out of the 'callable-iterator' objects that the inner generator expression is (I presume) yielding. But I may just be confused; I'm not much of a programmer (just someone who finds programming useful)!

Thanks for your time.

Answer 1

you have added an extra nesting of things to iterate over, but have not added an extra iteration to process the data. so code that was written to process a sequence of chunks is receiving a sequence of a sequence of chunks (one per line).

the code should look more like:

from itertools import chain

(chunk.group(1).strip(punctuation) 
 for chunk in chain.from_iterable(sep2.finditer(line) for line in file))

or (at this level of nesting it makes more sense to be explicit - it is very hard to see the problem in your "all in one line" code, but here it works "automatically" because you avoid creating a nested generator):

for line in file:
    for chunk in sep2.finditer(line):
        yield chunk.group(1).strip(punctuation)