How to iterate over a big Oracle database using python and jdbc-driver and store modified record values in another table?

Question

I have an Oracle DB with over 5 Million rows with columns of type varchar and blob. In order to connect to the database and read the records I use python 3.6 with a JDBC driver and the library JayDeBeApi. What I am trying to achieve is to read each row, perform some operations on the records (use a regex for example) and then store the new record values in a new table. I don't want to load all records in the memory, so what I want to do is to consequently fetch them from the database, store the fetched data, process it and then add it to the other table. Currently I fetch all the records at once instead for example first 1000, then the next 1000 and so on. This is what I have so far:

statement = "... a select statement..."
connection= dbDriver.connect(jclassname,[driver_url,username,password],jars,)
cursor = connection.cursor()
cursor.execute(statement)
fetched = cursor.fetchall()
for result in fetched:
    preprocess(result)
cursor.close()

How could I modify my code to fetch consequently and where to put the second statement which inserts the new values in the other table?

Answer 1

As you said, fetchall() is a bad idea in this case, as it loads all the data into the memory.

In order to avoid that you can iterate over cursor object itself:

cur.execute("SELECT * FROM test")
for row in cur:  # iterate over result set row by row
    do_stuff_with_row(row)
cur.close()