如何将标题添加到 csv 表 scala spark

Question

我正在尝试从csv文件中的表中读取数据。 它没有标题，所以当我尝试使用 Spark SQL 查询表时，所有结果都为空。

我曾尝试创建一个架构结构，虽然它在我执行printschema()时确实显示，但当我尝试并且（ select * from tableName ）它不起作用时，所有值都为空。 我也尝试过StructType()和.add( colName )而不是StructField并且产生了相同的结果。

        val schemaStruct1 = StructType(
            StructField( "AgreementVersionID", IntegerType, true )::
            StructField( "ProgramID", IntegerType, true )::
            StructField( "AgreementID", IntegerType, true )::
            StructField( "AgreementVersionNumber", IntegerType, true )::
            StructField( "AgreementStatusID", IntegerType, true )::
            StructField( "AgreementEffectiveDate", DateType, true )::
            StructField( "AgreementEffectiveDateDay", IntegerType, true )::
            StructField( "AgreementEndDate", DateType, true )::
            StructField( "AgreementEndDateDay", IntegerType, true )::
            StructField( "MasterAgreementNumber", IntegerType, true )::
            StructField( "MasterAgreementEffectiveDate", DateType, true )::
            StructField( "MasterAgreementEffectiveDateDay", IntegerType, true )::
            StructField( "MasterAgreementEndDate", DateType, true )::
            StructField( "MasterAgreementEndDateDay", IntegerType, true )::
            StructField( "SalesContactName", StringType, true )::
            StructField( "RevenueSubID", IntegerType, true )::
            StructField( "LicenseAgreementContractTypeID", IntegerType, true )::Nil
        )

        val df1 = session.read
            .option( "header", true )
            .option( "delimiter", "," )
            .schema( schemaStruct1 )
            .csv( LicenseAgrmtMaster )
        df1.printSchema()
        df1.createOrReplaceTempView( "LicenseAgrmtMaster" )

Printing this schema gives me this schema which is correct

root
 |-- AgreementVersionID: integer (nullable = true)
 |-- ProgramID: integer (nullable = true)
 |-- AgreementID: integer (nullable = true)
 |-- AgreementVersionNumber: integer (nullable = true)
 |-- AgreementStatusID: integer (nullable = true)
 |-- AgreementEffectiveDate: date (nullable = true)
 |-- AgreementEffectiveDateDay: integer (nullable = true)
 |-- AgreementEndDate: date (nullable = true)
 |-- AgreementEndDateDay: integer (nullable = true)
 |-- MasterAgreementNumber: integer (nullable = true)
 |-- MasterAgreementEffectiveDate: date (nullable = true)
 |-- MasterAgreementEffectiveDateDay: integer (nullable = true)
 |-- MasterAgreementEndDate: date (nullable = true)
 |-- MasterAgreementEndDateDay: integer (nullable = true)
 |-- SalesContactName: string (nullable = true)
 |-- RevenueSubID: integer (nullable = true)
 |-- LicenseAgreementContractTypeID: integer (nullable = true)

这是正确的，但是尝试查询这会给我一个只产生空值的表，即使该表没有填充空值。 我需要能够读取此表才能加入另一个表以完成存储过程

Answer 1

我建议按照以下步骤进行，然后您可以根据需要更改代码

val df = session.read.option( "delimiter", "," ).csv("<Path of your file/dir>")
val colum_names = Seq("name","id")// this is example define exact number of columns
val dfWithHeader = df.toDF(colum_names:_*)
// now you have header here and data should be also here check the type or you can cast