Spark：创建一个csv文件（必须使用scala和dataframe）

Question

I'm learning scala and dataframe recently and I came up a problem. It is about dataframe things. It must be solved using Scala and Dataframe , but NOT SparkSQL .

Problem:

1. Create a csv file with 4 columns (person, class, subject, score)for a school and put some random data into the csv, each person must have "Maths", "English", "Art" and plus some other subjects, and at least there are 3 classes.

2. Write a Spark program to:

   • read a csv file

   • show the full data table

   • show how many persons per class

   • show the person and his score with the highest score in "Maths"

I have tried to solve it and googled it, but what I came up is about using SQL to resolve it and also SQL is the first solution given by google.

I really don't know how to do it via Spark and Dataframe but NOT SparkSQL, though the tutorial said it was a very easy question:(

Could anyone help me with it, like write an example for me or give me an example? thank you so much. I will very appreciate it.

Answer 1

Sample csv file :

+-------+-------+---------+-------+   
| name  | class | subject | marks |
+-------+-------+---------+-------+
| ab    | 12    | Maths   | 72    |
+-------+-------+---------+-------+
| abc   | 12    | Maths   | 88    |
+-------+-------+---------+-------+
| abcd  | 11    | Arts    | 92    |
+-------+-------+---------+-------+
| abcde | 12    | English | 88    |
+-------+-------+---------+-------+
| bc    | 11    | Maths   | 99    |
+-------+-------+---------+-------+
| bcd   | 12    | English | 55    |
+-------+-------+---------+-------+
| bcde  | 11    | English | 77    |
+-------+-------+---------+-------+
| axax  | 10    | Maths   | 83    |
+-------+-------+---------+-------+
| amam  | 10    | English | 65    |
+-------+-------+---------+-------+
| arar  | 10    | Arts    | 66    |
+-------+-------+---------+-------+

---

1. Read csv file:

val df = spark.read.option("inferSchema","true").option("header","true").csv(filePath)

2. Show dataframe :
   df.show()

3. Show how many persons per class :
   df.groupBy("class").count.show

4. Show the person and his score with the highest score in "Maths": df.filter(col("subject")==="Maths").orderBy(desc("marks")).limit(1).show
   Moreover for the last question we can also filter out the class also.