计算 Dataframe Pyspark 中的行数

Question

I want to make some checks on my DF, in order to try it I'm using the following code:

start = '2020-12-10'
end = datetime.date.today()
country='gb'


df_ua = (spark.table(f'nn_squad7_{country}.fact_table')
      .filter(f.col('date_key').between(start,end))
      #.filter(f.col('is_client')==1)
      .filter(f.col('source')=='tickets')
      .filter(f.col('subtype')=='trx')
      .filter(f.col('is_trx_ok') == 1) 
      .select('ticket_id').distinct() 
      )

output = df_ua.count('ticket_id').distinct()

I'm getting the following error:

TypeError: count() takes 1 positional argument but 2 were given

I don't understand why I'm getting it, any clue?

Answer 1

Just doing df_ua.count() is enough, because you have selected distinct ticket_id in the lines above.

df.count() returns the number of rows in the dataframe. It does not take any parameters, such as column names. Also it returns an integer - you can't call distinct on an integer.

Answer 2

maybe you can try this instead:

import pyspark.sql.functions as f

start = '2020-12-10'
end = datetime.date.today()
country = 'gb'


df_ua = (spark.table(f'nn_squad7_{country}.fact_table')
      .filter(f.col('date_key').between(start, end))
      #.filter(f.col('is_client')==1)
      .filter(f.col('source')=='tickets')
      .filter(f.col('subtype')=='trx')
      .filter(f.col('is_trx_ok') == 1) 
      .select('ticket_id').distinct() 
      )

output = df_ua.count()