如何用spark python对CSV文件列求和

Question

i'm new in spark, and i have some data to work with, i want to sum the total of a column in a CSV file, the header of the file : ([colmun1],[colmun2],[colmun3]), what i'm trying to calculate is the sum of column3 according to column1 , (column1 is represent the date, column2 represent categorie,column3 the occurrence of one of the categories on that date, so i want to calculate the sum of all categories for each date), i have tried this code:

    from pyspark import SparkContext, SparkConf
    if __name__ == "__main__":
        conf = SparkConf().setAppName("sum").setMaster("local[3]")
        sc = SparkContext(conf = conf)
        line.split(",")).map(lambda line: (line[0:1]+line[3:4]))
        text_file = sc.textFile("in/fileinput.CSV")
        counts = text_file.flatMap(lambda line: line.split(",")) \
             .map(lambda line: (line[0:1],line[2:3])) \
             .reduceByKey(lambda a, b: a + b)
        counts.saveAsTextFile("out/fileoutput.txt")

thank you in advance (excuse my English)

Answer 1

Please try below steps to achieve the desired result.

1. Read the CSV file as Dataframe.

   df = spark.read.csv("path_to_csv_file", header=True, inferSchema=True)

2. Group By Data based on column 1.

   group_df = df.groupBy("Column_1")

3. Take sum of 3rd column on grouped data

   result_df = group_df.agg(sum("column_3").alias("SUM"))

4. Display Data result_df.show()

Hope it helps.

Note : for more information in CSV function Refer below link. https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.DataFrameReader.csv

Regards,

Neeraj