I'm working with scala to transform different java types to Spark DataType, so I wrote the following method:
def convertDataType(x : Class[_]): DataType = {
x match {
case i: Class[Integer] => IntegerType
case s: Class[String] => StringType
case d: Class[Double] => DoubleType
case l: Class[Long] => LongType
case s: Class[Short] => ShortType
case b: Class[Byte] => ByteType
case f: Class[Float] => FloatType
case ab: Class[Array[Byte]] => BinaryType
case bl: Class[Boolean] => BooleanType
case ts: Class[Timestamp] => TimestampType
case dt: Class[Date] => DateType
case _ => StringType
}}
However, with an input x (java.lang.String), it breaks out with a IntegerType. I tried another way:
def convertDataType(x : Class[_]): DataType = {
val S = classOf[String]
val I = classOf[Integer]
val D = classOf[Double]
val L = classOf[Long]
val SH = classOf[Short]
val B = classOf[Byte]
val F = classOf[Float]
val AB = classOf[Array[Byte]]
val BL = classOf[Boolean]
val TS = classOf[Timestamp]
val DT = classOf[Date]
x match {
case I => IntegerType
case S => StringType
case D => DoubleType
case L => LongType
case SH => ShortType
case B => ByteType
case F => FloatType
case AB => BinaryType
case BL => BooleanType
case TS => TimestampType
case DT => DateType
case _ => StringType
}}
The second way worked well and return the StringType as expected. I have no idea what's wrong in the first implementation?
Thanks a lot.
Matching : Class[Double]
is exactly the same as matching : Class[_]
because the generic type parameters aren't available. The compiler actually should warn you about it:
warning: non variable type-argument Double in type pattern java.lang.Class[Double] is unchecked since it is eliminated by erasure
This also happens for any other generic type except Array
s. There are a lot of Stack Overflow questions and articles about this, search for "type erasure".
In the second case, you are doing something entirely different: comparing values instead of checking types. This doesn't run into the above problem.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.