使用 Apache Spark 2.4 从在 SQL Server 2016 中存储为二进制（序列化哈希表）的 Spark 数据帧列获取哈希表/映射

Question

In one of the Legacy Application, In DB - SQL Server 2016, We have a Table - Measures

it has 15+ Columns, one of the column is binary

When I load into Spark and print the Schema, its binary

scala> jdbcDF.printSchema()
root
 |-- measurementValues: binary (nullable = true)
 |-- measure: string (nullable = true)

Looks like, they have used Hashtable , serialized it and stored into Table Column as binary

---

I am trying to de-serialize the same back into Hashtable (or) Map (or) Some Collection to able to convert into JSON Format while doing ETL Operations

Can anyone help here? I tried to convert the binary to string, still of no use :(

val convertToString = udf((a: Array[Byte])=> new String(a))

Answer 1

  def deserializeBinary = udf((x: Array[Byte]) => {
    val stream: InputStream  = new ByteArrayInputStream(x);
    val obs = new ObjectInputStream(stream)
    val stock = obs.readObject.asInstanceOf[util.Hashtable[String, String]]
    stock
  })