如何从火花数据帧scala中的地图中删除键

Question

I am working on data frame spark. There are 60 columns in my data frame and below is the sample map column of data frame. Need to remove 'N/A' key from map. I haven't find any function to do this

+----------------------------------+
| userbytestsample                 |
+----------------------------------+
|[TEST  -> 2000050008, N/A ->]     |
+----------------------------------+


schema  
 |-- userbytestsample: map (nullable = true)
 |    |-- key: string
 |    |-- value: string (valueContainsNull = true)

output

 +----------------------------------+
 | userbytestsample                 |
 +----------------------------------+
 |[TEST  -> 2000050008]             |
 +----------------------------------+

Answer 1

I suggest you use UDF

Let's suppose i have this dataframe

val df = Seq((Map("a" -> 1, "b" -> 2, "c" -> 3)), (Map("a" -> 10, "ff" -> 2, "gg" -> 30))).toDF("colmap")

scala> df.printSchema
root
 |-- colmap: map (nullable = true)
 |    |-- key: string
 |    |-- value: integer (valueContainsNull = false)

df.show(false)
+----------------------------+
|colmap                      |
+----------------------------+
|[a -> 1, b -> 2, c -> 3]    |
|[a -> 10, ff -> 2, gg -> 30]|
+----------------------------+

If i want to remove the key "a"

val unwantedKey : String = "a"

I create my UDF which will take the column 'colmap', remove the key and return the map without the key

def updateMap(unwantedKey : String) = udf((colMapName :Map[String, Int]) => {
  colMapName.-(unwantedKey)
})

Finally, to apply this udf you can call it this way

val finalDF = df.withColumn("newcol", updateMap(unwantedKey)(col("colmap")))
finalDF.show(false)
+----------------------------+-------------------+
|colmap                      |newcol             |
+----------------------------+-------------------+
|[a -> 1, b -> 2, c -> 3]    |[b -> 2, c -> 3]   |
|[a -> 10, ff -> 2, gg -> 30]|[ff -> 2, gg -> 30]|
+----------------------------+-------------------+

Answer 2

For Spark 3+ you can use map_filter as

df.select(map_filter($"userbytestsample", (k, v) => !k.equalTo("N/A")).as("userbytestsample"))
  .show(false)

Output:

 +----------------------------------+
 | userbytestsample                 |
 +----------------------------------+
 |[TEST  -> 2000050008]             |
 +----------------------------------+

For Spark 2.4+ you might need an udf

val map_filter_udf = udf{ (xs: Map[String, String]) => xs.filter(!_._1.equalsIgnoreCase(("N/A"))}

df.select(map_filter_udf($"userbytestsample"). as("userbytestsample"))
  .show(false)

Output:

 +----------------------------------+
 | userbytestsample                 |
 +----------------------------------+
 |[TEST  -> 2000050008]             |
 +----------------------------------+