如何更改 pyspark aws-glue 中的架构？

Question

I have a problem with a schema in pyspark. I am trying to read a complete collection from MongoDB whith glue dynamic frame.

    read_mongo_options = {
   "uri": mongo_uri,
   "database": "database",
   "collection": collection,
   "ssl": "true",
   "username": "user",
   "password": "password",
   "sampleSize": "200000",
   "partitioner": "MongoSamplePartitioner",
   "partitionerOptions.partitionSizeMB": "10",
   "partitionerOptions.partitionKey": "_id"

}
datasource0 = glueContext.create_dynamic_frame.from_options(connection_type="mongodb",                   
                                                          connection_options=read_mongo_options,transformation_ctx = "datasource0")
df=datasource0.toDF()

I think it is not able to interpret a schema that has different values in the same field. This is a value of defaultPaymentsource in an example;

  "defaultPaymentsource": {
    "$oid": "58f0933bXXXXXXXXXX"
  },

And in other documents, this fields doesn't exist anymore After inferring the schema, this is the result:

When I try to save or show results it gives me the following error

Cannot cast UNDEFINED into a StructType(StructField(oid,StringType,true))

I have set the value of samplesize to the maximum and I have changed the schema by removing the reference to 'undefined' but the result is always the same. Any Idea?

Answer 1

Well, it works now.

Dropping column an create ney df

    df_no_p=DropFields.apply(a, paths=['defaultPaymentsource.undefined'])
    schema2=df_no_p.toDF().schema
    df = spark.read.format("mongo").load(schema=schema2)