在火花中读取 dataframe 中的 XML 列

Question

I have following dataframe

+-----------+------+------------------------------------------------------------------
|ID         |xml                  |
+-----------+------+-----------------------------------------------------------------
|1          |<root><line><colX>1</colX></line><line><colX>2</colX></line></root>  |
|2          |<root><line><colX>3</colX></line><line><colX>4</colX></line> </root>
+-----------+------+-----------------------------------------------------------------

How do I convert it to following in raw spark sql using sparkXML from databricks

+-----------+------+------------------------------------------------------------------
|ID         |colx                  |
+-----------+------+-----------------------------------------------------------------
|1          | 1
 1            2 
|2          | 3
 2            3
+-----------+------+-----------------------------------------------------------------

Answer 1

You can use xpath to select the element into an array and explode the resulting array:

df2 = df.selectExpr('ID', "explode(xpath(xml, 'root/line/colX/text()')) as colx")

df2.show()
+---+----+
| ID|colx|
+---+----+
|  1|   1|
|  1|   2|
|  2|   3|
|  2|   4|
+---+----+