Bulk delete issue using sqlalchemy

Question

I am trying to delete a large number of records (up to 10,000) from a SQL Server 2016 DB in Microsoft Azure, based on a list of IDs, using the following code.

del_bool = Users.columns.ID.in_(idlist)
stmt_delete = Users.delete().where(del_bool)
results_proxy = connection.execute(stmt_delete)

This works fine when I have a small number of IDs < 2000 but once it gets above 4000 I get the following error,

sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('07002', '[07002] [Microsoft][ODBC Driver 13 for SQL Server]COUNT field incorrect or syntax error (0) (SQLExecDirectW)')

I tried chunking the ID list into 2000 chunks but get the following error when it attempts to delete the second chunk,

sqlalchemy.exc.DBAPIError: (pyodbc.Error) ('HY000', '[HY000] [Microsoft][ODBC Driver 13 for SQL Server]Connection is busy with results for another command

I feel this must be a common problem and there must be a better way to do this perhaps by delaying subsequent execute statements with the chunking method or using a different method altogether. I'm pretty new to sqlalchemy. Any help would be much appreciated.

Answer 1

I seem to have solved it by looping through the chunked ID list and using connectionless execution

for del_chunk in delete_chunks:
    del_bool = Users.columns.ID.in_(del_chunk)
    stmt_delete = Users.delete().where(del_bool)
    results_proxy = engine.execute(stmt_delete)

Suprised that Microsoft Azure would have trouble with deleting 10,000 records. The table is not big.