scala: classOf[], exceptions and :: operator for lists weird behavior

Question

I have encountered some realy weird behavior in scala. i wrote a generic method that takes as an argument an error prone code, and a list of "valid exceptions", and it should execute the code, while retrying the code when a "valid exception" is thrown.

the method works great and i use it in several places.
but then, one of the method calls failed at compilation.
and the cause was the initialization of the list of exceptions.

i have tried it in the REPL, to make sure there is no other cause, and you can see the results for yourself:

me@my-lap:~$ scala -cp /path/to/maven/repository/org/apache/httpcomponents/httpclient/4.1.1/httpclient-4.1.1.jar:/path/to/maven/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/path/to/maven/repository/me/my-utils/1.0-SNAPSHOT/my-utils-1.0-SNAPSHOT.jar:/path/to/maven/repository/org/apache/httpcomponents/httpcore/4.1/httpcore-4.1.jar
Welcome to Scala version 2.9.2 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_17).
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.apache.http.conn.{HttpHostConnectException,ConnectTimeoutException}
import org.apache.http.conn.{HttpHostConnectException, ConnectTimeoutException}

scala> import me.util.exceptions.RetryException
import me.util.exceptions.RetryException

scala> val validEx = classOf[ConnectTimeoutException] :: classOf[RetryException] :: classOf[HttpHostConnectException] :: Nil
<console>:9: error: inferred type arguments [java.lang.Class[_ >: _1 with org.apache.http.conn.ConnectTimeoutException <: java.lang.Exception]] do not conform to method ::'s type parameter bounds [B >: java.lang.Class[_ >: org.apache.http.conn.HttpHostConnectException with me.util.exceptions.RetryException <: java.lang.Exception]]
       val validEx = classOf[ConnectTimeoutException] :: classOf[RetryException] :: classOf[HttpHostConnectException] :: Nil
                                                      ^

when initializing a list from all 3 exception types, the code fails with this weird error. so i tried to initialize with every subset of 2 exceptions from those 3:

scala> val validEx = classOf[ConnectTimeoutException] :: classOf[RetryException] :: Nil
validEx: List[java.lang.Class[_ >: me.util.exceptions.RetryException with org.apache.http.conn.ConnectTimeoutException <: java.lang.Exception]] = List(class org.apache.http.conn.ConnectTimeoutException, class me.util.exceptions.RetryException)

scala> val validEx = classOf[ConnectTimeoutException] :: classOf[HttpHostConnectException] :: Nil
validEx: List[java.lang.Class[_ >: org.apache.http.conn.HttpHostConnectException with org.apache.http.conn.ConnectTimeoutException <: java.io.IOException]] = List(class org.apache.http.conn.ConnectTimeoutException, class org.apache.http.conn.HttpHostConnectException)

scala> val validEx = classOf[RetryException] :: classOf[HttpHostConnectException] :: Nil
validEx: List[java.lang.Class[_ >: org.apache.http.conn.HttpHostConnectException with me.util.exceptions.RetryException <: java.lang.Exception]] = List(class me.util.exceptions.RetryException, class org.apache.http.conn.HttpHostConnectException)

and it worked! i also tried to crete the list of all 3 exception types with List() instead of using :: operator. and it also worked:

scala> val validEx = List(classOf[ConnectTimeoutException], classOf[RetryException], classOf[HttpHostConnectException])
validEx: List[java.lang.Class[_ >: org.apache.http.conn.HttpHostConnectException with me.util.exceptions.RetryException with org.apache.http.conn.ConnectTimeoutException <: java.lang.Exception]] = List(class org.apache.http.conn.ConnectTimeoutException, class me.util.exceptions.RetryException, class org.apache.http.conn.HttpHostConnectException)

BTW, the implementation of RetryException is:

class RetryException extends Exception {}

So why did it happened? is it a bug with scala's :: operator?
and why couldn't the compiler infer a better type than: List[java.lang.Class[_ >: org.apache.http.conn.HttpHostConnectException with me.util.exceptions.RetryException with org.apache.http.conn.ConnectTimeoutException <: java.lang.Exception]]
(my method excepts an argument of type: validExceptions: List[Class[_ <: java.lang.Throwable]] which is a much more concise type. i would except the compiler to figure out somethong like: List[Class[_ <: java.lang.Exception]]

(i'm running on ubuntu 12.04 64bit with Scala version 2.9.2 (Java7 oracle 1.7.0_17)

Answer 1

This simply looks like an inference problem.

Try to replace Nil with List.empty[Exception] :

val validEx = classOf[ConnectTimeoutException] :: classOf[RetryException] :: classOf[HttpHostConnectException] :: List.empty[Class[_ <: Exception]]

This should fix it.

Now let's see what happens in the original code:

val validEx = classOf[ConnectTimeoutException] :: classOf[RetryException] :: classOf[HttpHostConnectException] :: Nil

This is evaluated from right to left, because :: is right-associative.

---

So first, the compiler evaluates classOf[HttpHostConnectException] :: Nil . The inferred type is List[Class[HttpHostConnectException]] . Let's call the result of this expression tmp1: List[Class[HttpHostConnectException]]

---

Then the compiler evaluates classOf[RetryException] :: tmp1 . Type inference tries to unify the types on the left and on the right, and comes up with List[_ >: RetryException with HttpHostConnectException] . You might wonder why it does not infer List[Class[Exception]] . This is because Class is not covariant (it is invariant), so Class[HttpHostConnectException] and Class[RetryException] are not subtypes of Class[Exception] . Now you might also argue the compiler could have inferred the simpler (but less precise) type List[AnyRef] . But the compiler just tries to be as precise as possible, which is obviously pretty usefulll in general. But in this case it will pose a problem with the next step. So let's call the result of this expression tmp2: List[_ >: RetryException with HttpHostConnectException] and see what happens next.

---

Finally the compiler evaluates classOf[ConnectTimeoutException] :: tmp2 .

Here the signature of :: mandates that classOf[ConnectTimeoutException] be a subtype of RetryException with HttpHostConnectException (the type of the list's elements so far). In other words ConnectTimeoutException is required to extends both RetryException and HttpHostConnectException , which clearly is not the case, and thus you get a compile error

Answer 2

When you're calling the cons operator ( :: ) multiple times, the type inferencer tries to evaluate the correct type parameter of the List for you at each step.

After the first couple of cons opearation

classOf[RetryException] :: classOf[HttpHostConnectException] :: Nil

the inferenced type is not compatible with the third cons call.
The error message tells you that the above value has the inferenced type (for the list elements) of

java.lang.Class[_ >: org.apache.http.conn.HttpHostConnectException with me.util.exceptions.RetryException <: java.lang.Exception]

So it expects that additional prepended elements must be supertypes of the above.

You can verify this on the console by doing

val intermediateList = classOf[RetryException] :: classOf[HttpHostConnectException] :: Nil

And look at the result type, and then try to prepend

classOf[ConnectTimeoutException] :: intermediateList

To get the same error as in the OP

---

As @Régis Jean-Gilles suggested, if you create the List specifying that the first element you build upon is of your expected type

List.empty[Exception]

The compiler should have no problem.

The same happens when you use the List(..., ...) object factory, since type inference for a method call is performed one parameter list (set of parens) at a time.
This means that the compiler uses all the arguments to define the correct List type