SQLAlchemy中的联合声明声明

Question

I have multiple large csv files that I want to merge and store the final table in a database for using in Pandas in the future. I read them all using Pandas and store them as separate but similar tables into a Sqlite database.

I want to merge the rows (vertically) by a SQL string and then use them in pandas for more analyses. I was wondering if this is a good practice to go back and forth between pandas and SQL for when we deal with large files and have limited memory (16GB)?

Also my code gives me an error and I was unsure if there is a syntax issue or something more significant that I missing here.

from sqlalchemy.sql import text
engine = create_engine('sqlite:///C:\\master.db', echo=False)
string = text("""SELECT * INTO Flows FROM (select * from "f2007-08" UNION select * from "f2009-10")""")
engine.execute(string)

Answer 1

That's not the correct syntax for inserting the results of a query into another table.

It's INSERT INTO tablename SELECT...

string = text("""
    INSERT INTO Flows 
    select * from "f2007-08" 
    UNION 
    select * from "f2009-10" 
""")