在Spark中具有不同标头的DataFrame中导入多个csv

Question

i have multiple Csv which everyone ra variable like this:

cloudiness.csv

    +---+---+----------+-------------------+
    |_c0| ID|cloudiness|           datetime|
    +---+---+----------+-------------------+
    |  0|  3|       1.0|2013-11-08 00:00:00|
    |  1|303|       2.0|2013-11-08 00:00:00|
    |  2|306|       3.0|2013-11-08 00:00:00|

temperature.csv

    +---+---+-----------+-------------------+
    |_c0| ID|temperature|           datetime|
    +---+---+-----------+-------------------+
    |  0|  3|        3.0|2013-11-08 00:00:00|
    |  1|303|        4.0|2013-11-08 00:00:00|
    |  2|306|        5.0|2013-11-08 00:00:00|

..and so on, (7 o 8 o this files).

I have to merge them in one only DataFrame using Spark (R, Python or Scala) like this:

    +---+---+-----------+----------+-------------------+
    |_c0| ID|temperature|cloudiness|           datetime|
    +---+---+-----------+----------+-------------------+
    |  0|  3|        3.0|       1.0|2013-11-08 00:00:00|
    |  1|303|        4.0|       2.0|2013-11-08 00:00:00|
    |  2|306|        5.0|       3.0|2013-11-08 00:00:00|

I tried spark.read but it takes too much time, files are 3 GB each one. What is the best method to do it?

Answer 1

The standard way is to join data frames.

when you read csv files using below snippet

val read_csv1 = sc.textFile("HDFS Path to read the file")

RDD will be created and you can join with other CSV's. If you mention performance issue means. let me give you another way.