Datastream Troubleshoot: "An unknown error occurred. Please try again. If the error persists, contact Google support"

Question

We are trying to replicate data from AlloyDB to Bigquery using Datastream. We Get "An unknown error occurred. Please try again. If the error persists, contact Google support."

In the Datastream console --> objects list, we see all source tables with Object Status "Failed" and Backfill status "Completed". In Bigquery we see only a subset of the tables (not all the "Completed" objects were synced).

In the Logs Explorer I can see this error on BQ:

I also see this error: error: { code: 11 message: "Unsupported primary key column either does not exist or is a pseudocolumn at [1:401]" } The column referred in the error is of type enum.

The desired situation is having all the AlloyDB tables replicated into Bigquery.

The error message is not very informative... What does it mean? What would be the best way to go about troubleshooting this?

Answer 1

I just had this same issue but connecting to a PostgreSQL in AWS RDS:

Beginning with Postgres 10, passwords are encrypted using SCRAM-SHA-256 in PostgreSQL. Google DataStream still expects MD5 password encryption, or it will generate an "unknown error" in the logs and fail the backfills.

You'll need to update your postgresql.conf (or RDS Cluster Parameter Group if you're using AWS like me):

password_encryption = 'MD5'

Restart the database and make sure the parameter has changed with:

SHOW password_encryption;

Reset the password of your users:

ALTER USER "{username}" with password '{password}';

More info from the PostgreSQL docs: https://www.postgresql.org/docs/current/auth-password.html

Answer 2

We're actively working on making these error messages be more informative, and improvements are continuously being rolled out as we identify more edge cases. Assuming you followed all the steps in the documentation , then you may need to open a ticket with support for further investigation. If a support ticket isn't an option, you can still report the issue using the public issue tracker