flask sqlalchemy update sql raw command does not provide any response

Question

I'm trying to perform an update using flask-sqlalchemy but when it gets to the update script it does not return anything. it seems the script is hanging or it is not doing anything.

I tried to wrap a try catch on the code that does not complete but there are no errors.

I gave it 10 minutes to complete the update statement which only updates 1 record and still, it will not do anything for some reason.

When I cancel the script, it provides an error Communication link failure (0) (SQLEndTran) but I don't think this is the root cause of the error because on the same script, I have other sql scripts that works ok so the connection to db is good

what my script does is get some list of filenames that I need to process (I have no issues with this). then using the retrieved list of filenames, I will look into the directory to check if the file exists. if it does not exists, I will update the database to tag the file as it is not found. this is where I get the issue, it does not perform the update nor provide an error message of some sort.

I even tried to create a new engine just for the update script, but still I get the same behavior.

I also tried to print out the sql script first in python before executing. I ran the printed sql command on my sql browser and it worked ok.

The code is very simple, I'm not really sure why it's having the issue.

#!/usr/bin/env python3

from flask_sqlalchemy import sqlalchemy
import glob


files_directory = "/files_dir/"

sql_string = """
    select * 
    from table
    where status is null
    """

# ommited conn_string
engine1 = sqlalchemy.create_engine(conn_string)
result = engine1.execute(sql_string)

for r in result:
    engine2 = sqlalchemy.create_engine(conn_string)
    filename = r[11]
    match = glob.glob(f"{files_directory}/**/{filename}.wav")

    if not match:
        print('no match')

        script = "update table set status = 'not_found' where filename = '" + filename + "' "
        engine2.execute(script)
        engine2.dispose()

    continue

engine1.dispose()

Answer 1

it appears that if I try to loop through 26k records, the script doesn't work. but when I try to do by batches of 2k records per run, then the script will work. so my sql string will become (added top 2000 on the query)

sql_string = """
    select top 2000 * 
    from table
    where status is null
    """

it's manual yeah, but it works for me since I just need to run this script once. (I mean 13 times)