如何为所有列编写withColumnRenamed并在Spark数据帧的自定义分区中加入两个不同的架构

Question

Hi i have two text file and i have to join this two text file to create unique one . I have used data frame in spark to achieve that .

Both text file has same structure except some fields .

Now i have to create data frame and join both data frame .

Question 1: How do we join both data frame that has some extra fields . for example my schema has first filed as TimeStamp but my first dataFrame does not have TimeStamp field .

Question 2: In my code i have to rename all column in order to select column after join and i have 29 columns so i have to write rename function 29 times .Is there any way i can do that without writing so many times .

Question 3: After Joining i have to save output as based on some filed . For example if StatementTypeCode is BAL then all records belonging to BAL will go to one file like that,same as custom partition in map reduce .

this is what i have tried latestForEachKey.write.partitionBy("StatementTypeCode") i hope it should be correct ..

I know i have asked so many question in one post .I am learning spark scala so facing issue in every syntax and every concept . I hope my question is clear .

Here is my code for what i am doing right now .

    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
        import sqlContext.implicits._

        import org.apache.spark.{ SparkConf, SparkContext }
        import java.sql.{Date, Timestamp}
        import org.apache.spark.sql.Row
        import org.apache.spark.sql.types.{ StructType, StructField, StringType, DoubleType, IntegerType,TimestampType }
        import org.apache.spark.sql.functions.udf

       val schema = StructType(Array(

    StructField("TimeStamp", StringType),
    StructField("LineItem_organizationId", StringType),
    StructField("LineItem_lineItemId", StringType),
    StructField("StatementTypeCode", StringType),
    StructField("LineItemName", StringType),
    StructField("LocalLanguageLabel", StringType),
    StructField("FinancialConceptLocal", StringType),
    StructField("FinancialConceptGlobal", StringType),
    StructField("IsDimensional", StringType),
    StructField("InstrumentId", StringType),
    StructField("LineItemLineItemName", StringType),
    StructField("PhysicalMeasureId", StringType),
    StructField("FinancialConceptCodeGlobalSecondary", StringType),
    StructField("IsRangeAllowed", StringType),
    StructField("IsSegmentedByOrigin", StringType),
    StructField("SegmentGroupDescription", StringType),
    StructField("SegmentChildDescription", StringType),
    StructField("SegmentChildLocalLanguageLabel", StringType),
    StructField("LocalLanguageLabel_languageId", StringType),
    StructField("LineItemName_languageId", StringType),
    StructField("SegmentChildDescription_languageId", StringType),
    StructField("SegmentChildLocalLanguageLabel_languageId", StringType),
    StructField("SegmentGroupDescription_languageId", StringType),
    StructField("SegmentMultipleFundbDescription", StringType),
    StructField("SegmentMultipleFundbDescription_languageId", StringType),
    StructField("IsCredit", StringType),
    StructField("FinancialConceptLocalId", StringType),
    StructField("FinancialConceptGlobalId", StringType),
    StructField("FinancialConceptCodeGlobalSecondaryId", StringType),
    StructField("FFFFAction", StringType)))


       val textRdd1 = sc.textFile("s3://trfsdisu/SPARK/Main.txt")
        val rowRdd1 = textRdd1.map(line => Row.fromSeq(line.split("\\|\\^\\|", -1)))
        var df1 = sqlContext.createDataFrame(rowRdd1, schema).drop("index")

        val textRdd2 = sc.textFile("s3://trfsdisu/SPARK/Incr.txt")
        val rowRdd2 = textRdd2.map(line => Row.fromSeq(line.split("\\|\\^\\|", -1)))
        var df2 = sqlContext.createDataFrame(rowRdd2, schema)

        // df2.show(false) 

        import org.apache.spark.sql.expressions._
        val windowSpec = Window.partitionBy("LineItem_organizationId", "LineItem_lineItemId").orderBy($"TimeStamp".cast(TimestampType).desc) 

        val latestForEachKey = df2.withColumn("rank", rank().over(windowSpec)).filter($"rank" === 1).drop("rank", "TimeStamp")
        .withColumnRenamed("StatementTypeCode", "StatementTypeCode_1").withColumnRenamed("LineItemName", "LineItemName_1").withColumnRenamed("FFAction", "FFAction_1")

  //This is where i need help withColumnRenamed part 


    val df3 = df1.join(latestForEachKey, Seq("LineItem_organizationId", "LineItem_lineItemId"), "outer")
          .select($"LineItem_organizationId", $"LineItem_lineItemId",
            when($"StatementTypeCode_1".isNotNull, $"StatementTypeCode_1").otherwise($"StatementTypeCode").as("StatementTypeCode"),
            when($"LineItemName_1".isNotNull, $"LineItemName_1").otherwise($"LineItemName").as("LineItemName"),
            when($"FFAction_1".isNotNull, $"FFAction_1").otherwise($"FFAction").as("FFAction")).filter(!$"FFAction".contains("D"))

        df3.show()

Answer 1

模式部分可以这样解决

val df1 = sqlContext.createDataFrame(rowRdd1, new StructType(schema.tail.toArray))