How can I avoid PG::InFailedSqlTransaction when adding new columns?

Question

We are running two rails applications against the same database. When we deploy, we typically deploy to App A, then App B, restarting all rails processes during the deploy. App A runs on 7 servers with at least 20 processes connections to the database. App B runs on 4 servers with at least 8 connections to the database.

Today, when we deployed App A, we added a column to an existing table:

change_table :organizations do |t|
  t.integer :users_count, default: 0
end

We expected this to be fine: its new column on an existing table and it has a default. Shortly after the migration ran, a number of errors showed up, from both App A (before it was restarted) and App B (before it was deployed to).

These errors were:

FATAL ActiveRecord::StatementInvalid error: PG::InFailedSqlTransaction:
ERROR:  current transaction is aborted, commands ignored until end of 
transaction block

In the postgres log, I have 58 errors like this:

postgres[12283]: ERROR:  cached plan must not change result type
postgres[12283]: STATEMENT:  SELECT  "organizations".* FROM 
  "organizations" WHERE "organizations"."id" = $1 LIMIT $2

This repeats a number of times and goes away after all deploys have finished and all processes restarted.

It appeared that Rails bug #12330 and Rails PR 22170 addressed this in Rails 5.0, but I have that commit and am still seeing this error.

Relevant software versions

• Rails 5.0.2
• PG 0.19.0
• Postgres 9.5

One comment on Rails bug #12330 suggests that I have to add the columns with null defaults. Another suggests performing multiple deploys, one to disable prepare statements, then another to perform the migration and and re-enable prepared statements.

Is there away to avoid this? It clears up when we restart the servers, but I feel like I'm missing something - like only using nullable columns perhaps that would avoid these errors all together. This doesn't happen on every deploy and I don't know how to reproduce it - but this wasn't the first time it has happened.

Answer 1

You have changed table structure and thus what your prepared SELECT returns, because you use "organizations".* , with is returning all columns. PostgreSQL apparently doesn't support updating of prepared statements, so you need to either create new session (reconnect) or use DEALLOCATE to remove that prepared statement.

EDIT: You could also stop using SELECT * .