如何聚合 spark SQL 中將其作為新列的列的非重復計數？

Question

spark.sql(f""" 
          INSERT INTO {databaseName}.{tableName} 
          SELECT 
              '{runDate}'
            , client_id
            , COUNT(DISTINCT client_id) AS distinct_count_client_id
          FROM df """) 

所以說我有一個 client_id 列有重復值，我想有一個客戶 ID 的聚合不同計數列，我將如何在 pyspark 中實現它？ 上面的代碼不起作用。

Answer 1

您可以使用 HAVING。 試試這個代碼：

spark.sql(f""" 
      INSERT INTO {databaseName}.{tableName} 
      SELECT 
          '{runDate}'
        , client_id
        , COUNT(*) AS client_id
        group by client_id HAVING COUNT(client_id) > 1
      FROM df """) 

Answer 2

您可以使用size和collect_set函數來實現 count distinct function。

spark.sql(f""" 
          insert into {databaseName}.{tableName} 
          select 
              '{runDate}'
              ,client_id
              ,size(collect_set(client_id) over (partition by null)) as distinct_count_client_id
          from df
          """
)