Spark添加具有值形式的新列前一些列

Question

I have DataFrame like this:

+----------+---+
|   code   |idn|
+----------+---+
|   [I0478]|  0|
|   [B0527]|  1|
|   [C0798]|  2|
|   [C0059]|  3|
|   [I0767]|  4|
|   [I1001]|  5|
|   [C0446]|  6|
+----------+---+

And i want to add new column to DataFrame

+----------+---+------+
|   code   |idn| item |
+----------+---+------+
|   [I0478]|  0| I0478|
|   [B0527]|  1| B0527|
|   [C0798]|  2| C0798|
|   [C0059]|  3| C0059|
|   [I0767]|  4| I0767|
|   [I1001]|  5| I1001|
|   [C0446]|  6| C0446|
+----------+---+------+

Please help me do this!

Answer 1

使用[] ：

df.withColumn("item", df["item"][0])

Answer 2

So the problem will be evident if you look at the schema - the column you are trying to subset is not an array. So the solution is to .* expand the column.

df.select('code.*', 'idn')

Answer 3

python

import pandas as pd


array = {'code': [['I0478'],['B0527'], ['C0798'], ['C0059'], ['I0767'], ['I1001'], ['C0446']], 'idn':[0, 1, 2, 3, 4, 5, 6]}


df = pd.DataFrame(array)

df['item'] = df.apply(lambda row: str(row.code).lstrip('[').rstrip(']').strip("'").strip(), axis= 1)


print(df)

Answer 4

df.withColumn("item", df["code"][0])

如果“item”列是Array类型，如果是String的Struct，可能需要通过df.select("code").collect()[0]检查item元素的key，看key是什么(string ）它有。