Convert nested list comprehension to use itertools chain in python

Question

Inspired by the discussion in Making a flat list out of list of lists in Python I tried to convert this (getting files and folders modification times in some_directory):

c = [os.path.getmtime(item) for root, d, files in os.walk(some_directory)
        for item in [root] + map(lambda fi: os.path.join(root, fi), files)]

To use itertools.chain:

c = map(os.path.getmtime,
        itertools.chain.from_iterable([root] + map(lambda fi: join(root, fi), files)
                                      for root, d, files in os.walk(some_directory)))

but my profiling shows it to be slower plus does not look really elegant.

So how can I use chain in this case, that is how can I more elegantly (and faster) produce the intermediate lists ?

Or is there some other itertools function for my case ?

EDIT:

Hashed up a profiling script:

import timeit

repeat = 10

setup ="""
import itertools
import os

join = os.path.join
path = r'C:\Dropbox\eclipse_workspaces'
c = []
"""


print "Original             ", min(timeit.Timer("""[c.extend([join(root,dir) for dir in dirs] + [join(root,file) for file in files]) for root,dirs,files in os.walk(path)]""",
                       setup=setup).repeat(3, repeat))
print "For loop             ", min(timeit.Timer("""for root, d, files in os.walk(path):
    c.append(root)
    c.extend(join(root, fi) for fi in files)""",
                       setup=setup).repeat(3, repeat))
print "Comprehension        ", min(timeit.Timer('[item for r, d, f in os.walk(path) for item in [r] + map(lambda f: join(r, f), f)]',
                       setup=setup).repeat(3, repeat))
print "Comprehension + chain", min(timeit.Timer('[item for r, d, f in os.walk(path) for item in itertools.chain.from_iterable(([r], map(lambda fi: join(r, fi), f)))]',
                       setup=setup).repeat(3, repeat))
print "Itertools            ", min(timeit.Timer("""[j for j in itertools.chain.from_iterable([root] + map(lambda fi: join(root, fi), files)
                                      for root, d, files in os.walk(path))]""",
                       setup=setup).repeat(3, repeat))

seems there is no difference but I had some strange artifacts when profiling so I don't post any results. I am still interested in the fastest way that can be done, preferably using itertools

Answer 1

I don't think it makes sense to use chain here. chain is most useful when flattening is the whole point of the comprehension it's replacing. If you're doing more complicated stuff, then it's probably easier just to stick with a comprehension or generator expression. Or even unpack the structure into an explicit for loop, which might allow you to avoid things like the list concatenation in your current code:

c = []
for root, d, files in os.walk(some_directory):
    c.append(root)
    c.extend(join(root, fi) for fi in files)