[英]Pyspark + association rule mining: how to transfer a data frame to a format suitable for frequent pattern mining?
I am trying to use pyspark to do association rule mining.我正在尝试使用 pyspark 进行关联规则挖掘。 Let's say my data is like:假设我的数据是这样的:
myItems=spark.createDataFrame([(1,'a'),
(1,'b'),
(1,'d'),
(1,'c'),
(2,'a'),
(2,'c'),],
['id','item'])
But according to https://spark.apache.org/docs/2.2.0/ml-frequent-pattern-mining.html , the format should be:但根据https://spark.apache.org/docs/2.2.0/ml-frequent-pattern-mining.html ,格式应该是:
df = spark.createDataFrame([(1, ['a', 'b', 'd','c']),
(2, ['a', 'c'])],
["id", "items"])
So I need to transfer my data from vertical to horizontal and the lengths for all the ids are different.所以我需要将我的数据从垂直传输到水平,并且所有 id 的长度都不同。
How can I do this transfer, or is there another way to do it?我该如何进行这种转移,或者有其他方法可以做到吗?
Let your original definition of myItems
be valid.让您对myItems
的原始定义有效。 collect_list
will be helpful after you typically group
the dataframe by id.在您通常按 id 对数据collect_list
进行group
后, collect_list
会有所帮助。
>>> myItems=spark.createDataFrame([(1,'a'),
... (1,'b'),
... (1,'d'),
... (1,'c'),
... (2,'a'),
... (2,'c'),],
... ['id','item'])
>>> from pyspark.sql.functions import collect_list
>>> myItems.groupBy(myItems.id).agg(collect_list('item')).show()
+---+------------------+
| id|collect_list(item)|
+---+------------------+
| 1| [a, b, d, c]|
| 2| [a, c]|
+---+------------------+
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.