Execute multiple scripts in Python by passing the scripts names

Question

I have a total of 25-30 scripts that updates data on the server. Since every day new data gets generated, so every day I have to run these scripts manually in a sequence and update the database with new data. All these scripts contain 300+ lines of SQL queries.

Now I want to automate this task with Python, but not really sure how to do that. I have used some libraries in past to connect to SQL server and then define a cursor to execute certain queries - cur.execute(select * from abc)

But now I want to automate this task and run all scripts by passing just the script names. Like

cur.execute(sql_script_1)
cur.execute(sql_script_2)
cur.execute(sql_script_3)
.
.
.
cur.execute(sql_script_25)

In this way, in the end, I'll just have to run this .py file and it will automatically run all scripts in the given order. Can this be done somehow? Either in this way or some other way.

The main motive is to automate the task of running all scripts by just passing the names.

Answer 1

Your question is probably a bad one, kinda vague and no effort shown in terms of research or implementing it yourself. But it's possible I had a similar use case, so I will share.

In my case I needed to pull down data as a pandas df. Your case might vary but I imagine basic structure will remain same.

In any case here is what I did:

1. you store each of the sql scripts as a string variable in a python file (or files) somewhere in your project directory.

2. you define a function that manages the connection and passes a sql script as an argument.

3. you call that function as needed for each script.

So your python file in one looks something like:

sql_script_1 = 'select....from...where...'
sql_script_2 = 'select....from...where...'
sql_script_3 = 'select....from...where...'

Then you define the function in two to manage the connection.

Something like

import pandas as pd
import pyodbc

def query_passer(query:str, conn_db):
    conn = pyodbc.connect(conn_db)
    df = pd.read_sql(query, conn)
    conn.close()
    return df

Then in three you call the function, do whatever you were gonna do with the data, and then repeat for each query.

The code below assume the query_passer function was saved in a python file named "functions.py" in the subfolder "resources" and that the queries are stored in a file named "query1.py" in the subfolder "resources.queries". Organize your own files as you wish, or just keep them all in one big file.

from resources.functions import query_passer
from resources.queries.query1 import sql_script_1, sql_script_2, sql_script_3
import pandas as pd

# define the connection 
conn_db = (
                "Driver={stuff};"
                "Server=stuff;"
                "Database=stuff;"
                ".....;"
)
# run queries
df = query_passer(sql_script_1, conn_db)
df.to_csv('sql_script_1.csv')

df = query_passer(sql_script_2, conn_db)
df.to_csv('sql_script_2.csv')

df = query_passer(sql_script_3, conn_db)
df.to_csv('sql_script_3.csv')