Using Python to import CSV from remote machine into Azure SQL Server

Question

I am trying to find a way to import a CSV file into Azure SQL Server from a remote machine using Python. I wrote a script to GET data from a third-party API and then flatten JSON into a CSV file. It works great. Now, I'm trying to improve the script to import that CSV into Azure SQL Server so that I don't have to manually upload the file using the Import Flat File tool.

with pyodbc.connect(f'''DRIVER={dw_driver};SERVER={dw_server};PORT={dw_port};DATABASE={dw_database};UID={dw_user};PWD={dw_password}''') as dw_connection:

    with dw_connection.cursor() as dw_curs:

        with open(out_file, 'r') as csv_file:
            for row in csv_file:
                dw_curs.execute('insert into DataOperations.StageTable (Id, CustomerName, CustomerEmail, CustomerOrganization, AssistedSalesPerson) values (%s, %s, %s, %s, %s)', row)

This code returns the following error:

ProgrammingError: ('The SQL contains 0 parameter markers, but 1 parameters were supplied', 'HY000')

When I print the generated SQL statement, it looks this:

insert into DataOperations.StageTable (Id, CustomerName, CustomerEmail, CustomerOrganization, AssistedSalesPerson) values (%s, %s, %s, %s, %s) 12345678|Test Name|testemail@test.com| Test Company |

Can anyone help me modify this script into a working SQL insert statement? I've trawled other similar questions on here but most recommend usin BULK INSERT which will not work because this file is not available on the database server.

Answer 1

The csv.reader() function must be used to create an object that can be used to iterate over lines in the target CSV file. The SQL parameters within the values() statement must use ? instead of %s to be valid syntax. The solution below is confirmed as working.

with pyodbc.connect(f'''DRIVER={dw_driver};SERVER={dw_server};PORT={dw_port};DATABASE={dw_database};UID={dw_user};PWD={dw_password}''') as dw_connection:

    with dw_connection.cursor() as dw_curs:

        with open(out_file, 'r', encoding='utf-8') as csv_file:
            next(csv_file, None)  # skip the header row
            reader = csv.reader(csv_file, delimiter='|')
            for row in reader:
                dw_curs.execute('insert into DataOperations.StageTable (Id, CustomerName, CustomerEmail, CustomerOrganization, AssistedSalesPerson) values (?, ?, ?, ?, ?)', row)