[英]pyspark: grouby and then get max value of each group
我想按值分組,然后使用PySpark在每個組中找到最大值。 我有以下代碼,但現在我有點不知道如何提取最大值。
# some file contains tuples ('user', 'item', 'occurrences')
data_file = sc.textData('file:///some_file.txt')
# Create the triplet so I index stuff
data_file = data_file.map(lambda l: l.split()).map(lambda l: (l[0], l[1], float(l[2])))
# Group by the user i.e. r[0]
grouped = data_file.groupBy(lambda r: r[0])
# Here is where I am stuck
group_list = grouped.map(lambda x: (list(x[1]))) #?
返回類似於:
[[(u'u1', u's1', 20), (u'u1', u's2', 5)], [(u'u2', u's3', 5), (u'u2', u's2', 10)]]
我想現在為每個用戶找到最大'發生'。 執行max后的最終結果將導致RDD看起來像這樣:
[[(u'u1', u's1', 20)], [(u'u2', u's2', 10)]]
只保留文件中每個用戶的最大數據集。 換句話說,我想將RDD的值更改為僅包含每個用戶最多出現的一個三元組。
這里不需要groupBy
。 簡單的reduceByKey
就可以了,並且大部分時間都會更有效:
data_file = sc.parallelize([
(u'u1', u's1', 20), (u'u1', u's2', 5),
(u'u2', u's3', 5), (u'u2', u's2', 10)])
max_by_group = (data_file
.map(lambda x: (x[0], x)) # Convert to PairwiseRD
# Take maximum of the passed arguments by the last element (key)
# equivalent to:
# lambda x, y: x if x[-1] > y[-1] else y
.reduceByKey(lambda x1, x2: max(x1, x2, key=lambda x: x[-1]))
.values()) # Drop keys
max_by_group.collect()
## [('u2', 's2', 10), ('u1', 's1', 20)]
我想我找到了解決方案:
from pyspark import SparkContext, SparkConf
def reduce_by_max(rdd):
"""
Helper function to find the max value in a list of values i.e. triplets.
"""
max_val = rdd[0][2]
the_index = 0
for idx, val in enumerate(rdd):
if val[2] > max_val:
max_val = val[2]
the_index = idx
return rdd[the_index]
conf = SparkConf() \
.setAppName("Collaborative Filter") \
.set("spark.executor.memory", "5g")
sc = SparkContext(conf=conf)
# some file contains tuples ('user', 'item', 'occurrences')
data_file = sc.textData('file:///some_file.txt')
# Create the triplet so I can index stuff
data_file = data_file.map(lambda l: l.split()).map(lambda l: (l[0], l[1], float(l[2])))
# Group by the user i.e. r[0]
grouped = data_file.groupBy(lambda r: r[0])
# Get the values as a list
group_list = grouped.map(lambda x: (list(x[1])))
# Get the max value for each user.
max_list = group_list.map(reduce_by_max).collect()
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.