spark createdataframe 无法推断架构 - 默认数据类型？

Question

Creating a spark dataframe in databricks using createdataframe results in error: 'Some of types cannot be determined after inferring'

I know I can specify the schema but that doesn't help if I'm creating the dataframe each time with source data from an API and they decide to restructure it.

Instead I'd like to tell spark to use 'string' for any column where a data type cannot be inferred.

Is this possible?

Thanks

Answer 1

This can be easily handled with schema evaluation with delta format. Quick ref: https://databricks.com/blog/2019/09/24/diving-into-delta-lake-schema-enforcement-evolution.html