pyspark 计算具有两个条件的行（AND 语句）

Question

I have the following code:

from pyspark.sql import functions as sf
data = [{"Category": 'Category A', "ID": 1, "Value": 12.40},
    {"Category": 'Category B', "ID": 2, "Value": 30.10},
     {"Category": 'Category C', "ID": 3, "Value": 100.01}
     ]
df = spark.createDataFrame(data)
print(df.schema)
df.show()
df.groupBy().agg(sf.count(sf.when(sf.col("Value")>13, True))).show()

This code gives:

+----------+---+------+
|  Category| ID| Value|
+----------+---+------+
|Category A|  1|  12.4|
|Category B|  2|  30.1|
|Category C|  3|100.01|
+----------+---+------+


+-------------------------------------------+
|count(CASE WHEN (Value > 13) THEN true END)|
+-------------------------------------------+
|                                          2|
+-------------------------------------------+

Which gives the total count of Values greater than 13. However, I want to find the total count of values greater than 13 and less than 100. This answer is '1'. The code

df.groupBy().agg(sf.count(sf.when(sf.col("Value")>13, True)),sf.count(sf.when(sf.col("Value")<100,True))).show()

returns:

  +-------------------------------------------+--------------------------------------------+
  |count(CASE WHEN (Value > 13) THEN true END)|count(CASE WHEN (Value < 100) THEN true END)|
   +-------------------------------------------+--------------------------------------------+
  |                                          2|                                           2|
  +-------------------------------------------+--------------------------------------------+

This isn't correct, this is giving the number of counts greater than 13, which is '2', and the counts less than 100, which is also '2'. But its not combining the 'when' functions. I also tried:

 df.groupBy().agg(sf.count(sf.when(sf.col("Value")>13 & sf.col("Value")<100),True)).show()

gives an error: py4j.protocol.Py4JError: An error occurred while calling o134.and. Trace:

So what is the right code to use to apply the 'and' function and get a desired output of '1'

Answer 1

There are many ways to achieve this.

Will show you the one that resembles your code.

df.groupby(f.when(f.col('id')<5,f.lit('lessThan5')).when(f.col('id')>=5,f.lit('GreaterOrEquals to 5'))).count().show()

+------------------------------------------------------------------------------+-----+
|CASE WHEN (id < 5) THEN lessThan5 WHEN (id >= 5) THEN GreaterOrEquals to 5 END|count|
+------------------------------------------------------------------------------+-----+
|                                                                     lessThan5|    2|
|                                                          GreaterOrEquals to 5|    3|
+------------------------------------------------------------------------------+-----+

Hope it helps

Answer 2

I believe you just need to wrap the conditions in parenthesis, so instead of:

df.groupBy().agg(sf.count(sf.when(sf.col("Value")>13 & sf.col("Value")<100),True)).show()

You can do:

df \
.groupBy() \
.agg(
   sf.count(
     sf.when(((sf.col("Value")>13) & (sf.col("Value")<100)))
     ,True
   )
).show()