Simultaneous queries in SQL

Question

I have started writing a telegram bot, which is accessed by many users at the same time. The bot does numerous SQL queries simultaneously for each user, when the user is using the bot.

When multiple users are using the bot, the bot crashes due to the following error:

"psycopg2.ProgrammingError: no results to fetch"

I think this happens, when the bot does a query for one user, while also doing a different query for another user.

Example: the cursor has done an "INSERT INTO" for one user, while also trying to fetch the results for the second user from the same cursor.

Two simultaneous transactions:

FIRST:

cursor.execute('''INSERT INTO USER_DATA(USER_ID, TRIAL_START) VALUES (%s, %s) ''', (m1, trial_date,))
conn.commit()
cursor.close()

SECOND:

cursor = conn.cursor()
cursor.execute('''SELECT * FROM USER_DATA WHERE USER_ID = %s''', (m1,))
conn.commit()
result = cursor.fetchall()
cursor.close()

As I think, the cursors might have done the SELECT statement, then the INSERT statement at the same time, however, then it fetches the result for the second transaction (SELECT), which gives the error as the cursor has just done the INSERT statement.

Is there a possibility to handle such cases somehow?

Answer 1

Maybe you need a connection pool and threading and a queue.

You basically put a connection handler between program and database server.

Very simplified, it manages the connections to the database server and keeps them open so that you don't have to establish a (time and ressource consuming) connection every time. These connections are made available to you as a connection pool. If you need a connection, you take one from the pool. If no connection is available, you have to wait until one becomes free. If you used a connection and don't need it anymore, you give it back to the pool.

You can either manage it directly in your code (eg psycopg2.pool), on the database server (eg PgBouncer) or as a separate service in between.

To use the connections simultaneously, you could use eg multithreading or multiprocessing. But be careful what you do with it in your database.

The user requests could then be queued until they get processed.

Not sure, if that helps.

Connection pool:
https://www.psycopg.org/docs/pool.html
http://www.pgbouncer.org/

A brief PgBouncer overview:
https://betterprogramming.pub/database-connection-pooling-with-pgbouncer-d8766a8a2c85

If you don't know, what connection pools are, maybe this article helps:
https://pynative.com/psycopg2-python-postgresql-connection-pooling/