Sqlalchemy constructing incorrectly SQL Query when not Reloading dataframe from CSV in Python

Question

我正在使用 python3、SQLAlchemy 和 MariaDB 服务器。

我从 json 格式的 REST 服务器获取数据，然后将其解析为字典，然后解析为 Z251D2BBFE9A3B78BCEAZDC6 中的 Dataframe。

当我不将 Dataframe 保存为 CSV 格式然后像这样重新加载时，会出现错误：

   df.to_csv("temp_save.csv", index=False)
   df = pd.read_csv("temp_save.csv")

当前面的行被注释掉时，我收到以下错误：

(pymysql.err.ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '), (), (), 0, '2022-01-26T17:32:49Z', 29101, 1, 3, 2, '2022-01-25T17:32:49Z', '2' at line 1")
[SQL: INSERT INTO `TicketRequesters` (subject, group_id, department_id, category, sub_category, item_category, requester_id, responder_id, due_by, fr_escalated, deleted, spam, email_config_id, fwd_emails, reply_cc_emails, cc_emails, is_escalated, fr_due_by, id, priority, status .....

VALUES (%(subject_m0)s, %(group_id_m0)s, %(department_id_m0)s, %(category_m0)s, %(sub_category_m0)s, %(item_category_m0)s, %(requester_id_m0)s, %(responder_id_m0)s, %(due_by_m0)s, %(fr_escalated_m0)s, %(deleted_m0)s, %(spam_m0)s, %(email_config_id_m0)s, %(fwd_emails_m0)s, %(reply_cc_emails_m0)s, %(cc_emails_m0)s, %(is_escalated_m0)s, %(fr_due_by_m0)s, %(id_m0)s, %(priority_m0)s, %(status_m0)s, %(source_m0)s, %(created_at_m0)s, %(updated_at_m0)s, %(requested_for_id_m0)s, %(to_emails_m0)s, %(type_m0)s, %(description_text_m0)s, %(custom_fields_localidad_m0)s, %(custom_fields_hora_de_la_falla_m0)s, %(custom_fields_hubo_alguna_modificacin_en_el_firewall_o_en_su_pl_m0)s, %(custom_fields_el_incidente_presentado_corresponde_a_m0)s, %(custom_fields_client_type_m0)s, %(custom_fields_riesgos_del_cambio_o_caso_m0)s, %(custom_fields_solucin_del_caso_m0)s, %(custom_fields_estado_de_cierre_m0)s, %(custom_fields_numero_de_oportunidad_m0)s, %(custom_fields_cuales_son_sus_servicios_afectados_especificar_si_m0)s, %(custom_fields_numero_de_ticket_de_cambio_m0)s, %(custom_fields_cantidad_estimada_de_personas_o_departamentos_afe_m0)s, %(cu.....

如图所示，在 VALUES %()s 字段中，最后附加了“_m0”，我注意到该数字增长到我尝试更新插入的行数。

%(stats_created_at_m29)s, %(stats_updated_at_m29)s, %(stats_ticket_id_m29)s, %(stats_opened_at_m29)s, %(stats_group_escalated_m29)s, %(stats_inbound_count_m29)s, %(stats_status_updated_at_m29)s, %(stats_outbound_count_m29)s, %(stats_pending_since_m29)s, %(stats_resolved_at_m29)s, %(stats_closed_at_m29)s, %(stats_first_assigned_at_m29)s, %(stats_assigned_at_m29)s, %(stats_agent_responded_at_m29)s, %(stats_requester_responded_at_m29)s, %(stats_first_responded_at_m29)s, %(stats_first_resp_time_in_secs_m29)s, %(stats_resolution_time_in_secs_m29)s, %(description_m29)s, %

这是我尝试使用的 python 代码，以防万一。

engine = db.create_engine(
    f"mariadb+pymysql://{user}:{password}@{host}/{database_name}?charset=utf8mb4"
)
columndict: dict = {"id": Column("id", Integer, primary_key=True)}

# Prepare Column List, check columndict if exists, get Column object from dict
column_list = [columndict.get(name, Column(name, String(256))) for name in columns]

# Get an instance of the table
# TODO: Instance table from metadata (without having to give it columns)
instanceTable = Table(table_name, metadata, *column_list)

metadata.create_all(engine)

# Schema created
# Create Connection
conn = engine.connect()

# Prepare statement
insert_stmt = db.dialects.mysql.insert(instanceTable).values(values)
on_duplicate_key_stmt = insert_stmt.on_duplicate_key_update(
    data=insert_stmt.inserted, status="U"
)
# Execute statement
result = conn.execute(on_duplicate_key_stmt)
# DATA UPSERTED

我调查了mysql/mariadb使用UTF8编码的局限性，正确的编码是?charset=utf8mb4，这可能与查询构造问题有关。

编辑：我通过将 Dataframe 中的空列表和空字符串替换为 None 找到了解决此错误的方法

Answer 1

此问题是由于向 SQLAlchemy 值发送空列表 [] 和空字符串 '' 引起的。

通过用无替换这些项目来修复。