Spark：删除 map 列的键

Question

I have a dataset ds containing a map column userInfo that has 3 keys:

{"name":"Tom", "age":"52", "phone":"45124"}

Now I want in my new dataset dsNew also have this map column (eg dsNew = ds.withColumn(...) ), but removes the key "phone". What is the best way performance-wise to do this?

Note that this is only a simplified example, in reality this map column has many key-value pairs and I want to remove a subset of it.

Thanks.

Answer 1

In addition to my linked answers which uses UDF, if your Spark version >= 3.0, you can also use map_filter :

dsNew = ds.withColumn("userInfo", expr("map_filter(userInfo, (k, v) -> k != 'phone')"))

If you have a number of keys to remove, you can use not in :

dsNew = ds.withColumn("userInfo", expr("map_filter(userInfo, (k, v) -> k not in ('phone', 'phone2'))"))

Answer 2

If you have only 3 keys it's simple to create new map by getting the 2 keys you want to keep:

val dsNew = ds.withColumn(
  "userInfo",
  map(lit("name"), $"userInfo"("name"), lit("age"), $"userInfo"("age"))
)

dsNew.show
//+------------------------+
//|userInfo                |
//+------------------------+
//|[name -> Tom, age -> 52]|
//+------------------------+

---

Map filtering is only available since version 3 of spark as pointed in the other answer. In spark 2.4, you can get the keys and filter them using array functions then create new map with the filtered keys using map_from_arrays function:

val dsNew = ds.withColumn(
    "filtered_keys",
    expr("filter(map_keys(userInfo), x -> x <> 'phone')")
  ).withColumn(
    "userInfo",
    map_from_arrays(
      $"filtered_keys",
      expr("transform(filtered_keys, x -> userInfo[x])")
    )
  ).drop("filtered_keys")