如何规避错误 pq_flush: could not send data to client: Broken pipe found

Question

I am trying to create an asynchronous process using Lambda functions that would look somewhat like this:

• Lambda 1 fires a query on my redshift cluster and ends
• Lambda 2 polls the cluster for the query status and would end/succeed based on results

I have used several different options, but they all seem to fail at one point. I can create a query, and fire it, and have the lambda end, but when the query completes execution instead of succeeding it complains that the client connection no longer exists

error pq_flush: could not send data to client: Broken pipe found in xyz

The problem is that this is completely expected for my use case. I don't want the client (Lambda 1) to wait around because my query could take an hour to run (exaggerating but possible) which is why I created a second lambda. Is there a way I can communicate this to Redshift/postgresql and circumvent this issue?

Here is my triggering code (will eventually go to lambda but I am testing on my local machine)

import select
import psycopg2

def wait(conn):
  while True:
    state = conn.poll()
    if state == psycopg2.extensions.POLL_OK:
      break
    elif state == psycopg2.extensions.POLL_WRITE:
      select.select([], [conn.fileno()], [])
    elif state == psycopg2.extensions.POLL_READ:
      select.select([conn.fileno()], [], [])
    else:
      raise psycopg2.OperationalError("poll() returned %s" % state)

conn = psycopg2.connect(
  user='someuser',
  dbname='somedb',
  host='myredshiftcluster',
  port=5432,
  password='somepassword',
  async_=1,
  sslmode="require"
)
wait(conn)
acurs = conn.cursor()
acurs.execute('call public.test_sp(\'xyz\')')

Answer 1

ODBC and JDBC connections are synchronous so building an asynchronous process around then will not work. Luckily AWS announced Redshift Data API recently which is an asynchronous REST interface. So you can perform what you are looking for through that method.

See: https://docs.aws.amazon.com/redshift/latest/mgmt/data-api.html