Passing a list of values from Python to the IN clause of an SQL query

Question

I am trying to pass a list like below to a sql query

x = ['1000000000164774783','1000000000253252111']

I am using sqlalchemy and pyodbc to connect to sql:

import pandas as pd
from pandas import Series,DataFrame
import pyodbc
import sqlalchemy

cnx=sqlalchemy.create_engine("mssql+pyodbc://Omnius:MainBrain1@172.31.163.135:1433/Basis?driver=/opt/microsoft/sqlncli/lib64/libsqlncli-11.0.so.1790.0")

I tried using various string format functions to be used in the sql query. below is one of them

  xx = ', '.join(x)
  sql = "select * from Pretty_Txns where Send_Customer in (%s)" % xx
  df = pd.read_sql(sql,cnx)

All of them seem to convert it into a numeric format

(1000000000164774783, 1000000000253252111) instead of ('1000000000164774783','1000000000253252111')

And hence the query fails as datatype of Send_Customer is varchar(50) in SQL

 ProgrammingError: (pyodbc.ProgrammingError) ('42000', '[42000] [Microsoft][SQL Server Native Client 11.0]
  [SQL Server]Error converting data type varchar to numeric. (8114) (SQLExecDirectW)') 
 [SQL: 'select * from Pretty_Txns where Send_Customer in (1000000000164774783, 1000000000253252111)']

Answer 1

As stated in the comment to the other answer, that approach can fail for a variety of reasons. What you really want to do it create an SQL statement with the required number of parameter placeholders and then use the params= parameter of read_sql to supply the values:

x = ['1000000000164774783','1000000000253252111']
placeholders = ','.join('?' for i in range(len(x)))  # '?,?'
sql = "select * from Pretty_Txns where Send_Customer in (%s)" % placeholders
df = pd.read_sql(sql, cnx, params=x)

Answer 2

Used the below approach and it worked fine:

       sql = "select * from Pretty_Txns where Send_Customer in %s" % str(tuple(x))
      df = pd.read_sql(sql,cnx)

Answer 3

Making sqlalchemey, pyodbc to work with pandas read_sql() is a hairy and messy thing. After much frustration and bumping into various solutions and documentation from pandas and psycopg , here's the correct (so far) way to do a query with named parameter:

import pandas as pd
import psycopg2
# import pyodbc 
import sqlalchemy
from sqlalchemy import text # this is crucial

cnx=sqlalchemy.create_engine(...)
x = ['1000000000164774783','1000000000253252111']
sql = "select * from Pretty_Txns where Send_Customer in (:id);" # named parameter
df = pd.read_sql(text(sql), cnx, params={'id':x}) # note how `sql`
                                                   # string is cast with text() 
                                                   # and key-pair value is passed for 
                                                   # named parameter 'id'
df.head()

I've made it work with the PostgreSQL database. I hope it wouldn't be too much different for MySQL.

Answer 4

Here's the SQL query you need

sql = f"select * from Pretty_Txns where Send_Customer in {tuple(x)}"
df = pd.read_sql(sql,cnx)