Scala for comprehensions and flatMap / map intermediate results

Question

From the first time I read that:

for {
  harpo<-list1 if harpo.length>0
  groucho<-list2
  chico<-list3
} yield (harpo, groucho, chico)

translates to:

list1.filter(_.length>0).flatMap(harpo =>      
      list2.flatMap(groucho=>list3.map((harpo,groucho,_)))
)

I was worried about unnecessary intermediate collections returned by filter , flatMap & map . The first was fixed in Scala 2.8(?) by addition of the withFilter method, and I suspect that there's some magic going on which changes the return type of these methods depending on usage so when used as a parameter to flatMap they return a non-strict collection, but I can't find any proof. Are my suspicions right and it isn't as ineffective as it seems by first glance?

Answer 1

This relates to this question . Specifically, the answer by @IODEV shows you how to look at the desugared form:

$ scala -Xprint:typer -e
'val list1 = List("foo", "bar"); val list2 = list1; val list3 = list1;
for (harpo<-list1 if harpo.length>0; groucho <- list2; chico <- list3) 
yield (harpo, groucho, chico)'

(without line breaks)

list1.withFilter(_.length() > 0)
  .flatMap(harpo =>
    list2.flatMap(groucho =>
      list3.map(chico => (harpo, groucho, chico))
    )
  )

---

I don't see any wasted intermediate collections that you could save, unless you go for a mutable builder and while or foreach calls to fill that builder:

val b = List.newBuilder[(String, String, String)]
for(harpo <- list1 if harpo.length() > 0; groucho <- list2; chico <- list3) {
  b += ((harpo, groucho, chico))
}
b.result()

---

The question is, does your particular code exhibit a significant performance problem. Eg your collections are tremendously large. If not, go with the more idiomatic form ( for ... yield ). Only optimise to a builder with for ... {} when you actually gain something from it.