Check if dataframe satisfies database table column definitions

Question

Is there a way to check if pandas dataframe satisfies definitions of table comlumns in a database? I want to do this inspection before using pd.to_sql() funcion in order to avoid errors and exceptions.

Say for example that a table column is of VARCHAR(50) type, but one record in df has length of more than 50 and I want to catch this before pd.to_sql() execution. VARCHAR might actually one of the easier ones since it's just a string but my tables contain dates, numbers, etc.

I am able to obtain definitions using:

from sqlalchemy import inspect
inspection = inspect(engine)
definitions = inspection.get_columns('table')

# definitions contains a list of dicts like:
# [{'name': 'col_name', 'type': VARCHAR(length=255), 'default': None, 'nullable': True}, {...}, {...}]

but I am stuck at how to actually compare all values with definitions.

I am running multiple ETLs and doing this in a easy way is would be great.

Answer 1

在插入数据库之前，您可以使用Pydantic模型来解析和验证您的数据。