将 sql 语句转换为 pyspark

Question

I have created a sql code that I want to convert into pyspark code. Except for one thing, it works. But how can I best insert the sum function.

SELECT r_date, abc_code, sum(case when kpi_id=1234 then value else null end) as XXX, 
       sum(case when kpi_id=5678 then value else null end) as YYY from rate 
WHERE abc_code = 'AS55' AND org_id = '12-3' 
GROUP BY r_date, abc_code 
ORDER BY r_date DESC, abc_code;

Pyspark code

rate_df = rate_df.select(
    'org_id',
    'abc_code',
    'value',
    'r_date',
    expr("case when kpi_id == '1234' then value else ' ' end").alias('XXX'),
    expr("case when kpi_id == '5678' then value else ' ' end").alias('YYY')
    ) \
    .where((F.col('abc_code') == 'AS55') &
           (F.col('organisation_id') == '12-3'))

How can I insert the sum function in pyspark to get the values in one row.

Answer 1

The follow code I have implemented:

rate_df = rate_df.select(
    F.col('creation_date').alias('r_date'),
    'organisation_id',
    'b_employee',
    'abc_code',
    'kpi_date',
    'kpi_id',
    'value'
    )
nh_rate_df = rate_df.where(
        (F.col('abc_code') == 'AS55') &
        (F.col('organisation_id') == '12-3')
     ).groupBy(
        'organisation_id', 'r_date', 'b_employee', 'kpi_date', 'abc_code'
    ).agg(
        F.sum(F.when(F.col('kpi_id') == 1234, F.col('value'))).alias('xxx'),
        F.sum(F.when(F.col('kpi_id') == 5678, F.col('value'))).alias('YYY'),
    ).orderBy(
        F.desc('kpi_date'), F.col('abc_code')
    )
nh_rate_df = nh_rate_df.join(s_function_df, 'abc_code', 'left')
nh_rate_df = nh_rate_df.join(hst_df, 'organisation_id', 'left')

I get a result that doesn't summarize all of the matching lines.

r_date	kpi_date	organisation_id	abc_code	b_empl	XXX	YYY
2020-12-02	2020-11-01 00:00:00	12-3	AS55	A	1.0000	null
2020-12-02	2020-11-01 00:00:00	12-3	AS55	null	null	1.0000
2020-11-02	2020-10-01 00:00:00	12-3	AS55	A	null	1.0000
2020-11-02	2020-10-01 00:00:00	12-3	AS55	null	1.0000	null
2020-10-02	2020-09-01 00:00:00	12-3	AS55	A	2.0000	null
2020-10-02	2020-09-01 00:00:00	12-3	AS55	null	null	1.0000
2020-09-22	2020-08-01 00:00:00	12-3	AS55	null	1.0000	1.0000
2020-09-22	2020-08-01 00:00:00	12-3	AS55	A	null	null

although i have the same date. can it be due to the join?

This result Iam getting if I run the sql code

r_date	kpi_date	organisation_id	abc_code	b_empl	XXX	YYY
2020-12-02	2020-11-01 00:00:00	12-3	AS55	A	1.0000	2.0000
2020-11-02	2020-10-01 00:00:00	12-3	AS55	A	1.0000	2.0000
2020-10-02	2020-09-01 00:00:00	12-3	AS55	A	2.0000	1.0000
2020-09-22	2020-08-01 00:00:00	12-3	AS55	null	2.0000	1.0000