从 Pyspark Dataframe 中删除与 header 匹配的行

Question

我有一个巨大的 dataframe 类似于：

l = [('20190503', 'par1', 'feat2', '0x0'),
('20190503', 'par1', 'feat3', '0x01'),
('date', 'part', 'feature', 'value'),
('20190501', 'par5', 'feat9', '0x00'),
('20190506', 'par8', 'feat2', '0x00f45'),
('date', 'part', 'feature', 'value'),
('20190501', 'par11', 'feat3', '0x000000000'),
('date', 'part', 'feature', 'value'),
('20190501', 'par3', 'feat9', '0x000'),
('20190501', 'par6', 'feat5', '0x000000'),
('date', 'part', 'feature', 'value'),
('20190506', 'par8', 'feat1', '0x00000'),
('20190508', 'par3', 'feat6', '0x00000000'),
('20190503', 'par4', 'feat3', '0x0c0deffe21'),
('20190503', 'par6', 'feat4', '0x0000000000'),
('20190501', 'par3', 'feat6', '0x0123fe'),
('20190501', 'par7', 'feat4', '0x00000d0')]

columns = ['date', 'part', 'feature', 'value']


+--------+-----+-------+------------+
|    date| part|feature|       value|
+--------+-----+-------+------------+
|20190503| par1|  feat2|         0x0|
|20190503| par1|  feat3|        0x01|
|    date| part|feature|       value|
|20190501| par5|  feat9|        0x00|
|20190506| par8|  feat2|     0x00f45|
|    date| part|feature|       value|
|20190501|par11|  feat3| 0x000000000|
|    date| part|feature|       value|
|20190501| par3|  feat9|       0x000|
|20190501| par6|  feat5|    0x000000|
|    date| part|feature|       value|
|20190506| par8|  feat1|     0x00000|
|20190508| par3|  feat6|  0x00000000|
|20190503| par4|  feat3|0x0c0deffe21|
|20190503| par6|  feat4|0x0000000000|
|20190501| par3|  feat6|    0x0123fe|
|20190501| par7|  feat4|   0x00000d0|
+--------+-----+-------+------------+

它有与 header 匹配的行，我想删除所有行，这样结果将是：

+--------+-----+-------+------------+
|    date| part|feature|       value|
+--------+-----+-------+------------+
|20190503| par1|  feat2|         0x0|
|20190503| par1|  feat3|        0x01|
|20190501| par5|  feat9|        0x00|
|20190506| par8|  feat2|     0x00f45|
|20190501|par11|  feat3| 0x000000000|
|20190501| par3|  feat9|       0x000|
|20190501| par6|  feat5|    0x000000|
|20190506| par8|  feat1|     0x00000|
|20190508| par3|  feat6|  0x00000000|
|20190503| par4|  feat3|0x0c0deffe21|
|20190503| par6|  feat4|0x0000000000|
|20190501| par3|  feat6|    0x0123fe|
|20190501| par7|  feat4|   0x00000d0|
+--------+-----+-------+------------+

我试图用.distinct()方法摆脱它们，但总是留下一个。

我该怎么做？

Answer 1

这会起作用（本质上是链接多个过滤器，Spark 将在创建物理计划时负责合并它们）

for col in df.schema.names:
   df=df.filter(F.col(col) != col)

df.show()

输入：

Output：