Python loop list to append tables from sqlite3

Question

Within my sqlite3 database there are hundreds of tables, but what I would like to do is create a dataframe which appends only tables from the database that match the names contained in separate list that I have made

The list is called 'col_list' and currently contains only 3 elements (3 names)

col_list = df['ref_name'].tolist()

My attempt so far has lead me to the following, which is very cumbersome. :

conn = sqlite3.connect('all_data.db')
query = "SELECT * FROM " + col_list[0] + ";"
df = pd.read_sql_query(query, conn)

conn = sqlite3.connect('all_data.db')
query = "SELECT * FROM " + col_list[1] + ";"
df1 = pd.read_sql_query(query, conn)
df2 = df.append(df1)

conn = sqlite3.connect('all_data.db')
query = "SELECT * FROM " + col_list[2] + ";"
df3 = pd.read_sql_query(query, conn)
df4 = df2.append(df3)

df4 = df4.sort_values(by = 'date')
df4 = df4.reset_index(drop=True)

The number of elements in the 'col_list' can vary, which based on my current code structure means rewriting the code each time that this happens. Ultimately I would like to be able to have this all work as a 'for' loop and therefore look to you guys for help.

Thank you for taking the time to read this.

Answer 1

If I understood your question correctly, you want to do something like this?

df_all = None
conn = sqlite3.connect('all_data.db')
for col in col_list:
    query = "SELECT * FROM " + col + ";"
    df = pd.read_sql_query(query, conn)
    if df_all is not None:
        # See also @Parfait's comment below
        # about performance cost of append()
        df_all = df_all.append(df)
    else:
        df_all = df
conn.close()
df_all = df_all.sort_values(by = 'date')
df_all = df_all.reset_index(drop=True)