Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call? Best practices for using Knex.Transaction

Question

When working with a big application that has several tables and several DB operations it's very difficult to keep track of what transactions are occurring. To workaround this we started by passing around a trx object.

This has proven to be very messy.

For example:

async getOrderById(id: string, trx?: Knex.Transaction) { ... }

Depending on the function calling getOrderById it will either pass a trx object or not. The above function will use trx if it is not null.

This seems simple at first, but it leads to mistakes where if you're in the middle of a transaction in one function and call another function that does NOT use a transaction, knex will hang with famous Knex: Timeout acquiring a connection. The pool is probably full. Knex: Timeout acquiring a connection. The pool is probably full.

async getAllPurchasesForUser(userId: string) {
  ..
  const trx = await knex.transaction();
  try {
    ..
    getPurchaseForUserId(userId); // Forgot to make this consume trx, hence Knex timesout acquiring connection.
    ..
}

Based on that, I'm assuming this is not a best practice, but I would love if someone from Knex developer team could comment.

To improve this we're considering to instead use knex.transactionProvider() that is accessed throughout the app wherever we perform DB operations.

The example on the website seems incomplete:

// Does not start a transaction yet
const trxProvider = knex.transactionProvider();

const books = [
  {title: 'Canterbury Tales'},
  {title: 'Moby Dick'},
  {title: 'Hamlet'}
];

// Starts a transaction
const trx = await trxProvider();
const ids = await trx('catalogues')
  .insert({name: 'Old Books'}, 'id')
books.forEach((book) => book.catalogue_id = ids[0]);
await  trx('books').insert(books);

// Reuses same transaction
const sameTrx = await trxProvider();
const ids2 = await sameTrx('catalogues')
  .insert({name: 'New Books'}, 'id')
books.forEach((book) => book.catalogue_id = ids2[0]);
await sameTrx('books').insert(books);

In practice here's how I'm thinking about using this:

SingletonDBClass.ts:

const trxProvider = knex.transactionProvider();
export default trxProvider;

Orders.ts

import trx from '../SingletonDBClass';
..
async getOrderById(id: string) {
  const trxInst = await trx;
  try {
    const order = await trxInst<Order>('orders').where({id});
    trxInst.commit();
    return order;
  } catch (e) {
    trxInst.rollback();
    throw new Error(`Failed to fetch order, error: ${e}`);
  }
}
..

Am I understanding this correctly?

Another example function where a transaction is actually needed:

async cancelOrder(id: string) {
  const trxInst = await trx;
  try {
    trxInst('orders').update({ status: 'CANCELED' }).where({ id });
    trxInst('active_orders').delete().where({ orderId: id });
    trxInst.commit();
  } catch (e) {
    trxInst.rollback();
    throw new Error(`Failed to cancel order, error: ${e}`);
  }
}

Can someone confirm if I'm understanding this correctly? And more importantly if this is a good way to do this. Or is there a best practice I'm missing?

Appreciate your help knex team!

Answer 1

No. You cannot have global singleton class returning the transaction for your all of your internal functions. Otherwise you are trying always to use the same transaction for all the concurrent users trying to do different things in the application.

Also when you once commit / rollback the transaction returned by provider, it will not work anymore for other queries. Transaction provider can give you only single transaction.

Transaction provider is useful in a case, where you have for example middleware, which provides transaction for request handlers, but it should not be started, since it might not be needed so you don't want yet allocate a connection for it from pool.

Good way to do your stuff is to pass transcation or some request context or user session around, so that each concurrent user can have their own separate transactions.

for example:

async cancelOrder(trxInst, id: string) {
  try {
    trxInst('orders').update({ status: 'CANCELED' }).where({ id });
    trxInst('active_orders').delete().where({ orderId: id });
    trxInst.commit();
  } catch (e) {
    trxInst.rollback();
    throw new Error(`Failed to cancel order, error: ${e}`);
  }
}

Depending on the function calling getOrderById it will either pass a trx object or not. The above function will use trx if it is not null.

This seems simple at first, but it leads to mistakes where if you're in the middle of a transaction in one function and call another function that does NOT use a transaction, knex will hang with famous Knex: Timeout acquiring a connection. The pool is probably full.

We usually do it in a way that if trx is null, query throws an error, so that you need to explicitly pass either knex / trx to be able to execute the method and in some methods trx is actually required to be passed.

Anyhow if you really want to force everything to go through single transaction in a session by default you could create API modules in a way that for each user session you create an API instance which is initialized with transaction:

const dbForSession = new DbService(trxProvider);

const users = await dbForSession.allUsers();

and .allUsers() does something like return this.trx('users');