如何使用 Python 列表中的 SQLAlchemy 更新和/或插入 MySQL 数据库

Question

I am building a webscrape that will run over and over that will insert new data or update data based on ID. if 'id' == 'id': My goal is to avoid duplicates. MySQL table is ready and built. What is the best Pythonic way to check your python list before inserting/updating it in MySQL DB using SQLAlchemy?

Below are my dependenices:

from sqlalchemy.ext.automap import automap_base
from sqlalchemy.orm import Session 
import requests
from bs4 import BeautifulSoup
from time import sleep
from datetime import datetime
import time
engine = create_engine("mysql+pymysql:///blah")

I use a function to assign each <td> from scraped data:

def functionscrape( **kwargs ):
    scrape = {
        'id': '',
        'owner': '',
        'street': '',
        'city': '', 
        'state': '', 
        }
    scrape.update(kwargs)
    return (scrape)

The list below is an example, but would be changing constantly with each webscrape.

myList = 
[{
        'id': '111',
        'owner': 'Bob',
        'street': '1212 North',
        'city': 'Anywhere', 
        'state': 'TX', 
        },
{
        'id': '222',
        'owner': 'Mary',
        'street': '333 South',
        'city': 'Overthere', 
        'state': 'AZ', 
        }]

Answer 1

I am using a helper function to create the dynamic sql update queries:

def construct_update(table_name, where_vals, update_vals):
    query = table_name.update()
    for k, v in where_vals.items():
        query = query.where(getattr(table_name.c, k) == v)
    return query.values(**update_vals)

basically you pass the function the table and 2 dictionaries. The first would just be {'id': id} in your case, and the second is all the values you want to update, like

{
    'owner': 'Bob',
    'street': '1212 North',
    'city': 'Anywhere', 
     etc...
}

the helper function then returns the query which can be executed with

my_session = Session(engine)
my_session.execute(query)

Unfortunately, using this method, you'll have to update every single row individually (no bulk update) - but if you can live with that this works fine

otherwise here's a similar post about bulk updates: Bulk update in SQLAlchemy Core using WHERE

Answer 2

You can try using https://marshmallow.readthedocs.io/en/stable/ library to make validation

Build Schema and define fields with types you need. You can also use @pre_load and @post_load decorators to manipulate your data