如何更改现有 dataframe 的模式

Question

Problem statement: I have a csv file with around 100+ fields.I need to perform transformation on these fields and generate new 80+ fields and write only these new fields into s3 in parquet format.

The parquet predefined schema = 80+ newly populated fields + some non populated fields.

Is there any way to pass this predefined parquet schema while writing data to s3 so that these extra fields also populated with null data.

select will not be useful to select only 80+ fields as predefined schema might have around 120 predefined fields.

Below is sample data and transformation requirementCSV data

aid, productId, ts, orderId

1000,100,1674128580179,edf9929a-f253-487

1001,100,1674128580179,cc41a026-63df-410

1002,100,1674128580179,9732755b-1207-471

1003,100,1674128580179,51125ddd-4129-48a

1001,200,1674128580179,f4917676-b08d-41e

1004,200,1674128580179,dc80559d-16e6-4fa

1005,200,1674128580179,c9b743eb-457b-455

1006,100,1674128580179,e8611141-3e0e-4d5
1002,200,1674128580179,30be34c7-394c-43a

Parquet schema

def getPartitionFieldsSchema() = {
  List(
    Map("name" -> "company", "type" -> "long",
      "nullable" -> true, "metadata" -> Map()),
    Map("name" -> "epoch_day", "type" -> "long",
      "nullable" -> true, "metadata" -> Map()),
    Map("name" -> "account", "type" -> "string",
      "nullable" -> true, "metadata" -> Map()),
  )
}

val schemaMap = Map("type" -> "struct",
  "fields" -> getPartitionFieldsSchema)

simple example

val dataDf = spark
  .read
  .format("csv")
  .option("header", "true")
  .option("inferSchema", "true")
  .load("./scripts/input.csv")


dataDf
  .withColumn("company",lit(col("aid")/100))
  .withColumn("epoch_day",lit(col("ts")/86400))
  .write   // how to write only company, epoch_day, account ?
  .mode("append")
  .csv("/tmp/data2")

Output should have below columns: company, epoch_day, account

Answer 1

This is how I understand your problem: you wanna read some csv and transform them to parquet in s3. during the transformation, you need to create 3 new cols based on existing cols in csv files. but since only 2 out of 3 new cols are calculated, the output are only showing two new cols but not 3.

In such case, you can create a external table in redshift, and you specify all the cols. As a result even some colums are not fed, there would be null in your external tables.