在Spark中对JAVA进行SQL查询

Question

I am trying to convert a SQL query to Spark program with JAVA for practice. I am posting the schemas of the two files I am using. Also the query which I am trying to convert

Schema of each file: Store_return's schema

root
 |-- datetime: long (nullable = true)
 |-- sr_returned_date_sk: long (nullable = true)
 |-- sr_return_time_sk: long (nullable = true)
 |-- sr_item_sk: long (nullable = true)
 |-- sr_customer_sk: long (nullable = true)
 |-- sr_cdemo_sk: long (nullable = true)
 |-- sr_hdemo_sk: long (nullable = true)
 |-- sr_addr_sk: long (nullable = true)
 |-- sr_store_sk: long (nullable = true)
 |-- sr_reason_sk: long (nullable = true)
 |-- sr_ticket_number: long (nullable = true)
 |-- sr_return_quantity: integer (nullable = true)
 |-- sr_return_amt: double (nullable = true)
 |-- sr_return_tax: double (nullable = true)
 |-- sr_return_amt_inc_tax: double (nullable = true)
 |-- sr_fee: double (nullable = true)
 |-- sr_return_ship_cost: double (nullable = true)
 |-- sr_refunded_cash: double (nullable = true)
 |-- sr_reversed_charge: double (nullable = true)
 |-- sr_store_credit: double (nullable = true)
 |-- sr_net_loss: double (nullable = true)

date_dim's schema:

root
 |-- d_date_sk: long (nullable = true)
 |-- d_date_id: string (nullable = true)
 |-- d_date: string (nullable = true)
 |-- d_month_seq: integer (nullable = true)
 |-- d_week_seq: integer (nullable = true)
 |-- d_quarter_seq: integer (nullable = true)
 |-- d_year: integer (nullable = true)
 |-- d_dow: integer (nullable = true)
 |-- d_moy: integer (nullable = true)
 |-- d_dom: integer (nullable = true)
 |-- d_qoy: integer (nullable = true)
 |-- d_fy_year: integer (nullable = true)
 |-- d_fy_quarter_seq: integer (nullable = true)
 |-- d_fy_week_seq: integer (nullable = true)
 |-- d_day_name: string (nullable = true)
 |-- d_quarter_name: string (nullable = true)
 |-- d_holiday: string (nullable = true)
 |-- d_weekend: string (nullable = true)
 |-- d_following_holiday: string (nullable = true)
 |-- d_first_dom: integer (nullable = true)
 |-- d_last_dom: integer (nullable = true)
 |-- d_same_day_ly: integer (nullable = true)
 |-- d_same_day_lq: integer (nullable = true)
 |-- d_current_day: string (nullable = true)
 |-- d_current_week: string (nullable = true)
 |-- d_current_month: string (nullable = true)
 |-- d_current_quarter: string (nullable = true)
 |-- d_current_year: string (nullable = true)oss|

The query is

select sr_customer_sk as ctr_customer_sk
      ,sr_store_sk as ctr_store_sk
      ,sum(sr_return_quantity) as ctr_total_return
      from store_returns
      ,date_dim
      where sr_returned_date_sk = d_date_sk
      and d_year = 2003
      group by sr_customer_sk
      ,sr_store_sk

For the same, I have written the following uptil now

  Dataset<Row> df = store_returns
              .join(date_dim, store_returns.col("sr_returned_date_sk").equalTo(date_dim.col("d_date_sk")));

      df.groupBy("sr_customer_sk","sr_store_sk").agg(sum("sr_return_quantity").alias("ctr_total_return"))
              .select(col("sr_returned_date_sk").alias("ctr_customer_sk"),
                        col("sr_store_sk").alias("ctr_store_sk"))
        .where(col("d_year").equalTo("2003").and(col("sr_returned_date_sk").equalTo(col("d_date_sk"))))
          .groupBy("sr_customer_sk","sr_store_sk").agg(sum("sr_return_quantity").alias("ctr_total_return")).show();;

I am getting the following error with it

Exception in thread "main" 18/04/23 14:31:40 WARN Utils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf. org.apache.spark.sql.AnalysisException: cannot resolve ' sr_returned_date_sk ' given input columns: [sr_customer_sk, sr_store_sk, ctr_total_return];; 'Project ['sr_returned_date_sk AS ctr_customer_sk#309, sr_store_sk#8L AS ctr_store_sk#310L] +- Aggregate [sr_customer_sk#4L, sr_store_sk#8L], [sr_customer_sk#4L, sr_store_sk#8L, sum(cast(sr_return_quantity#11 as bigint)) AS ctr_total_return#304L] +- Join Inner, (sr_returned_date_sk#1L = d_date_sk#43L) :- Relation[datetime#0L,sr_returned_date_sk#1L,sr_return_time_sk#2L,sr_item_sk#3L,sr_customer_sk#4L,sr_cdemo_sk#5L,sr_hdemo_sk#6L,sr_addr_sk#7L,sr_store_sk#8L,sr_reason_sk#9L,sr_ticket_number#10L,sr_return_quantity#11,sr_return_amt#12,sr_return_tax#13,sr_return_amt_inc_tax#14,sr_fee#15,sr_return_ship_cost#16,sr_refunded_cash#17,sr_reversed_charge#18,sr_store_credit#19,sr_net_loss#20] parquet +- Relation[d_date_sk#43L,d_date_id#44,d_date#45,d_month_seq#46,d_week_seq#47,d_quarter_seq#48,d_year#49,d_dow#50,d_moy#51,d_dom#52,d_qoy#53,d_fy_year#54,d_fy_quarter_seq#55,d_fy_week_seq#56,d_day_name#57,d_quarter_name#58,d_holiday#59,d_weekend#60,d_following_holiday#61,d_first_dom#62,d_last_dom#63,d _same_day_ly#64,d_same_day_lq#65,d_current_day#66,... 4 more fields] parquet

Answer 1

df.groupBy("sr_customer_sk","sr_store_sk").agg(sum("sr_return_quantity").alias("ctr_total_return"))

This will result in dataframe with 3 columns sr_customer_sk , sr_store_sk , ctr_total_return on which select("sr_returned_date_sk") will not work because dataframe doesn't have sr_returned_date_sk .

Try using:

Dataset<Row> df = store_returns
              .join(date_dim, store_returns.col("sr_returned_date_sk").equalTo(date_dim.col("d_date_sk")))
.where(col("d_year").equalTo("2003"));

df.groupBy("sr_customer_sk","sr_store_sk").agg(sum("sr_return_quantity").alias("ctr_total_return"))
              .select(col("sr_customer_sk").alias("ctr_customer_sk"),
                        col("sr_store_sk").alias("ctr_store_sk"),col("ctr_total_return"))