將標簽字符串轉換為二進制向量 pyspark

Question

我有看起來像這樣的數據：

| Id | ----Tags---- | some_text |
| 0  | <a><b>       | ex1       |
| 1  | <a><c>       | ex2       |
| 2  | <b><c>       | ex3       |

我希望它最終看起來像這樣：

| Id | a | b | c | some_text |
| 0  | 1 | 1 | 0 | ex1       |
| 1  | 1 | 0 | 1 | ex2       |
| 2  | 0 | 1 | 1 | ex3       |

我想使用 pyspark 作為解決方案。 關於如何解決這個問題的任何想法？

Answer 1

If you don't already know the expected categorical values, you can use pyspark.sql.functions.udf to split and tags into an array of values and pyspark.sql.functions.explode function to convert them to columns. 然后，您可以將 pivot 值添加到列中：

# required imports
import pyspark.sql.functions as F
from pyspark.sql.types import ArrayType, StringType
import re

# regex pattern to split 'tagged values'
pat = re.compile('<(.*?)>')

#udf to split string to array of values
split_f = f.udf(lambda s: pat.split(s), ArrayType(StringType()))

# sample data
df = spark.createDataFrame([(0,'<a><b>','ex1'),(1,'<a><c>','ex2')], ['Id', '---Tags---', 'some_text'])

+---+----------+---------+
| Id|---Tags---|some_text|
+---+----------+---------+
|  0|    <a><b>|      ex1|
|  1|    <a><c>|      ex2|
+---+----------+---------+

df.withColumn('exploded', 
   F.explode(split_f(F.col('---Tags---'))))
  .groupby('Id').pivot('exploded').count().na.fill(0).show()

+---+---+---+---+
| Id|  a|  b|  c|
+---+---+---+---+
|  0|  1|  1|  0|
|  1|  1|  0|  1|
+---+---+---+---+