encoding issues generating schema definition in pyspark for dataframe

Question

I want to generate a schema definition from an XML file to generate a post deployment test for our notebooks.

I've gotten the XML parsed into the following string:

StructType([StructField('ItemNumber', StringType(), True),
            StructField('UPC', StringType(), True),
            StructField('AssignDate', DateType(), True),
            StructField('AssignmentQuantity', IntegerType(), True)]

and my data into the form:

[Row(dataRow="'A123456', '12345678900', '12/01/2020', 89"),
 Row(dataRow="'B123456','00123456789', 12/02/2018, 1002")]

this is the code:

# create a dataframe from mock test data
def CreateMockInputData(notebook_Name, entity_Name, dataSpec):
    schema = CreateEntitySchema(notebook_Name=notebook_Name, dataSpec=dataSpec, entity_Name=entity_Name)
    print(schema)
    # parse out the data
    entityDef = NotebookEntity(notebook_Name=notebook_Name, dataSpec=dataSpec, entity_Name=entity_Name)
    data_list = entityDef.selectExpr("explode(data_row) as dataRow").collect()
    print()
    print(data_list)
    entity_data = spark.createDataFrame(data_list, schema)
    return entity_data


mock_df = CreateMockInputData(notebook_Name='Test Notebook', dataSpec=df_entityDataDefinitions,
                              entity_Name='entity_for_data'))

What I'm getting is the following error:

ParseException                            Traceback (most recent call last)
<command-4322020421037787> in <module>()
----> 1 mock_df = CreateMockInputData(notebook_Name = 'Test Notebook', dataSpec = df_entityDataDefinitions, entity_Name = 'entity_for_data')
      2 #print(mock_df)
      3 mock_df.printSchema()
      4 mock_df.show(10, False)

<command-4322020421037786> in CreateMockInputData(notebook_Name, entity_Name, dataSpec)
     10   print()
     11   print(data_list)
---> 12   entity_data = spark.createDataFrame(data_list, schema)
     13   entity_data = entityData_list
     14   return entity_data

/databricks/spark/python/pyspark/sql/session.py in createDataFrame(self, data, schema, samplingRatio, verifySchema)
    735 
    736         if isinstance(schema, basestring):
--> 737             schema = _parse_datatype_string(schema)
    738         elif isinstance(schema, (list, tuple)):
    739             # Must re-encode any unicode strings to be consistent with StructField names​

It's unclear to me how or what needs to be "re-encoded" to get the schema to work with my data.

any suggests would be welcomed.

Answer 1

In order to convert a string defining a schema I found you need to execute the string using the eval statement.

example:

 schema_str = "StructType([StructField('ItemNumber', StringType(), True), StructField('UPC', StringType(), True), StructField('AssignDate', DateType(), True), StructField('AssignmentQuantity', IntegerType(), True)]" entity_data = spark.createDataFrame(data_list,eval(schema))