我如何比较 PySpark 中另一个 dataframe 的列

Question

I have two dataframes:
First with AVG values:

+----------+-----+  
| Category | AVG |  
+----------+-----+  
| Categ    | 1.0 |  
+----------+-----+  
| Categ2   | 0.5 | 
+----------+-----+
... 

...
Second has the fallowing category: Category, Name, Price
The question is: How can I delete all those records for which the price is less than the average price from the first table??
I tried that way:

dataGreaterAvge = data.where(data.Price >= avgCategoryPrice.where(data.Category == avgCategoryPrice.Category).collect()[0]["avg(Price)"])

dataGreaterAvge  - First dataframe
data - Second dataframe

However, this does not work as it should, because it only takes the value of the first element from the average values table

Answer 1

Spark works like SQL... so...

First you need to join the dataframes.

a = df1.alias('a')
b = df2.alias('b')
df_joined = a.join(b, a.Category == b.Category)

then you will be able to filter properly

from pyspark.sql import functions as f

df_joined.select(col('a.category'),col('a.AVG'))\
         .where(col('a.AVG') > f.avg(col('b.avg')).groupBy(col('a.AVG'))