Pyspark 如何计算每个组中字符串的出现次数并打印多个选定的列？

Question

My data set looks like this. Each row represents a car. Each car is located at an Auto Center , has a Model , Make , and a bunch of other attributes. This is a simplified version of the data frame. Extraneous rows and columns have been omitted for clarity.

+===========+========+=======+====+=====+
|Auto Center|Model   |Make   |Year|Color|
+===========+========+=======+====+=====+
|Roseville  |Corolla |Toyota |    |     |
|Roseville  |Prius   |Toyota |    |     |
|Rocklin    |Camry   |Toyota |    |     |
|Rocklin    |Forester|Subaru |    |     |
+===========+========+=======+====+=====+

What do I want to do? I want to group the data by the Auto Center , and display a "list" of the top 5 cars in each Auto Center by quantify, and print their attributes Make , Model , Year , and Color .

After grouping the data by the Auto Center , I want to count the number of occurrences of each Model , or even better a combination of Make and Model , in each Auto Center , and I want to get a list of the top 5 cars with the most occurrences. Then I want to print multiple columns of that car.

Assume that the Year and the Color are the same for each car having the same Make and Model .

For example, the output should be something like this, a list of the top 5 cars in each auto center ordered by the number of occurrences.

Rosevile:
there are 12 red Toyota Prius 2009
there are 8 blue Toyota Cary 2010
 ...

This is what I have so far:

from pyspark import SparkContext
from pyspark.sql import SQLContext, SparkSession
from pyspark.sql.types import StructType, StructField
from pyspark.sql.types import DoubleType, IntegerType, StringType

sc = SparkContext()
sqlContext = SQLContext(sc)
scSpark = SparkSession \
    .builder \
    .appName("Auto Center Big Data") \
    .config("spark.some.config.option", "some-value") \
    .getOrCreate()

data = scSpark.read.csv("autocenters.csv", header=True, inferSchema=True)
data.printSchema();

data.groupby('Auto Center')

It seems that data.groupby() returns a GroupedData object. I have seem that .agg() function can be applied to it, but that only works for numerical data, such as finding the mean of some numbers, and here I have strings. I want to count the strings by number of occurrences in each group.

What should I do? Is there a way to apply an aggregate function to multiple columns simultaneously, such as both Make and Model together? If not, that should be fine though, considering that there are no cars with the same Model having different Make s.

Answer 1

IIUC, you can do it with the following two steps:

1. First groupby all columns you want to count on the occurences:

    df1 = df.groupby('Auto Center', 'Model', 'Make', 'Year', 'Color').count()

2. Then set up a Window Spec and get the top 5 by row_number(): ( Note: depends on how you want to handle ties, you might want to change the function row_number() to rank() or dense_rank() )

    from pyspark.sql import Window from pyspark.sql.functions import row_number, desc w1 = Window.partitionBy('Auto Center').orderBy(desc('count')) df_new = df1.withColumn('rn', row_number().over(w1)).where('rn <= 5').drop('rn')

Answer 2

import pyspark

from pyspark.sql import *
from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession
sc = SparkContext('local')
spark = SparkSession(sc)

then create a random dataframe similar to yours

import string
import random

n = 1000
center = ['center_{}'.format(random.choice(string.ascii_letters[:2])) for y in range(n)]
make = ['make_{}'.format(random.choice(string.ascii_letters[:3])) for y in range(n)]
model = ['model_{}'.format(random.choice(string.ascii_letters[:3])) for y in range(n)]
year = [random.choice(range(2018, 2019)) for y in range(n)]
color = [random.choice(['red', 'blue', 'black']) for y in range(n)]

df = spark.createDataFrame(zip(center, make, model, year, color), schema=['center', 'make', 'model', 'year', 'color'])

df.head()

Row(center='center_b', make='make_a', model='model_c', year=2018, color='black')

group by center, make, model, year, color

df_groupby = (df
              .groupby('center', 'make', 'model', 'year', 'color')
              .count()
             )
df_groupby.sort(df_groupby['center'],
                df_groupby['count'].desc()).show()

+--------+------+-------+----+-----+-----+
|  center|  make|  model|year|color|count|
+--------+------+-------+----+-----+-----+
|center_a|make_c|model_b|2018| blue|   33|
|center_a|make_a|model_c|2018| blue|   24|
|center_a|make_a|model_a|2018|  red|   23|
|center_a|make_b|model_c|2018| blue|   21|
|center_a|make_b|model_b|2018|black|   21|
|center_a|make_c|model_a|2018|black|   21|
|center_a|make_c|model_c|2018| blue|   21|
|center_a|make_a|model_c|2018|black|   20|
|center_a|make_a|model_b|2018|  red|   20|
|center_a|make_a|model_b|2018| blue|   19|
|center_a|make_c|model_c|2018|black|   18|
|center_a|make_a|model_c|2018|  red|   18|
|center_a|make_c|model_b|2018|  red|   18|
|center_a|make_b|model_b|2018|  red|   18|
|center_a|make_c|model_a|2018|  red|   18|
|center_a|make_a|model_b|2018|black|   18|
|center_a|make_b|model_c|2018|black|   18|
|center_a|make_a|model_a|2018| blue|   17|
|center_a|make_c|model_a|2018| blue|   17|
|center_a|make_c|model_b|2018|black|   15|
+--------+------+-------+----+-----+-----+
only showing top 20 rows

using a Window, keep only the top 5 make/model/color/year per center

from pyspark.sql import Window
from pyspark.sql.functions import row_number, desc

w = Window.partitionBy('center').orderBy(desc('count'))
df_groupby2 = df_groupby.withColumn('rn', row_number().over(w)).where('rn <= 5').drop('rn')

df_groupby2.sort(df_groupby2['center'],
                 df_groupby2['count'].desc()
                ).show()

+--------+------+-------+----+-----+-----+
|  center|  make|  model|year|color|count|
+--------+------+-------+----+-----+-----+
|center_a|make_c|model_b|2018| blue|   33|
|center_a|make_a|model_c|2018| blue|   24|
|center_a|make_a|model_a|2018|  red|   23|
|center_a|make_b|model_b|2018|black|   21|
|center_a|make_c|model_a|2018|black|   21|
|center_b|make_a|model_a|2018|  red|   31|
|center_b|make_c|model_c|2018|black|   24|
|center_b|make_b|model_a|2018| blue|   24|
|center_b|make_b|model_b|2018|black|   23|
|center_b|make_c|model_c|2018| blue|   23|
+--------+------+-------+----+-----+-----+

now create and print your text

df_final = (df_groupby2
            .withColumn('text', F.concat(F.lit("there are "),
                                         df_groupby2['count'],
                                         F.lit(" "),
                                         df_groupby2['color'],
                                         F.lit(" "),
                                         df_groupby2['make'],
                                         F.lit(" "),
                                         df_groupby2['model'],
                                         F.lit(" "),
                                         df_groupby2['year'])
                       )
            .sort(df_groupby2['center'],
                 df_groupby2['count'].desc()
                )
           )

for row in df_final.select('center').distinct().sort('center').collect():
    current_center = row['center']
    print(current_center, ":")
    for row in df_final.filter(df_final['center'] == current_center).collect():
        print(row['text'])

center_a :
there are 33 blue make_c model_b 2018
there are 24 blue make_a model_c 2018
there are 23 red make_a model_a 2018
there are 21 black make_b model_b 2018
there are 21 black make_c model_a 2018
center_b :
there are 31 red make_a model_a 2018
there are 24 blue make_b model_a 2018
there are 24 black make_c model_c 2018
there are 23 black make_b model_b 2018
there are 23 blue make_c model_c 2018