Apache Spark join/cogroup on generic type RDD

Question

I have a problem with the join or cogroup methods on RDD . In detail, I have to join two RDD s and one of them is an RDD of a generic type, used with wildcard.

val indexedMeasures = measures.map(m => (m.id(), m)) // RDD[(String, Measure[_]]
val indexedRegistry = registry.map(r => (r.id, r))   // RDD[(String, Registry)]
indexedRegistry.cogroup(indexedMeasures)

The last statement gives a compile time error, which is the following:

no type parameters for method cogroup: (other: org.apache.spark.rdd.RDD[(String, W)])org.apache.spark.rdd.RDD[(String, (Iterable[Registry], 
 Iterable[W]))] exist so that it can be applied to arguments (org.apache.spark.rdd.RDD[(String, Measure[?0]) forSome { type ?0 }]) --- because --- argument expression's type is not compatible 
 with formal parameter type; found : org.apache.spark.rdd.RDD[(String, Measure[?0]) forSome { type ?0 }] required: org.apache.spark.rdd.RDD[(String, ?W)] Note: (String, 
 Measure[?0]) forSome { type ?0 } >: (String, ?W), but class RDD is invariant in type T. You may wish to define T as -T instead. (SLS 4.5)

What's going on here? Why can't I cogroup RDD s that use a generic wildcarded type?

Thanks for all of your responses.

Answer 1

The problem is stated in this article Towards Equal Rights for Higher-kinded Types

Generics are a very popular feature of contemporary OO languages, such as Java, C# or Scala. Their support for genericity is lacking, however. The problem is that they only support abstracting over proper types, and not over generic types. This limitation makes it impossible to, eg, define a precise interface for Iterable, a core abstraction in Scala's collection API. We implemented “type constructor polymorphism” in Scala 2.5, which solves this problem at the root, thus greatly reducing the duplication of type signatures and code.