在 spark 中将列转换为二进制数据类型

Question

I have a Spark use case where I have to create a null column and cast to a binary datatype. I tried the below but it is not working. When I replace Binary by integer, it works. I also tried BinaryType and Array[Byte]. Must be missing something here.

val ip1 = sqlContext.read
    .parquet("/home/hadoop/work/aa/bbb/ccc/data/today")
    .toDF();
val ip2 = ip1
    .withColumn("user_hll",lit("0"))
    .select(col("start_timestamp_hr"),col("user_hll"))
    .withColumn("hll",col("user_hll").cast("Binary"))

Any help is appreciated

Answer 1

如果你想要一个空二进制列，你只需要：

.withColumn("hll", lit(null).cast(org.apache.spark.sql.types.BinaryType))

Answer 2

instead of casting you could also use lit directly

.withColumn("hll",lit("0".getBytes)) // gives [30]

or

.withColumn("hll",lit(Array.empty[Bytes])) // gives []

or if you want ǹull , you can do :

.withColumn("hll",typedLit(Option.empty[Array[Byte]])) // gives null

Answer 3

It worked. Simple miss from my side

.cast("Binary") 

Any better way would be appreciated