Migration from Heroku PostgreSQL to AWS RDS using AWS Data Migration Service (DMS) not working

Question

I'm trying to use AWS DMS to copy and replicate data from a Heroku PostgreSQL database to an AWS RDS PostgreSQL instance but it isn't working (more infos below).

In DMS Log I can see the following message:

2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: RetCode: SQL_ERROR SqlState: 42501 NativeError: 1 Message: ERROR: must be superuser or replication role to use replication slots;, Error while executing the query [1022502] (ar_odbc_stmt.c:2421)

Heroku PostgreSQL's user is not a superuser or have a replication role but I've made an experiment with the same source database 2 days ago and it worked correctly (I stopped the experiment to create a bigger RDS instance target). I think I'm forgetting something in configuration this time.

• Source: Heroku PostgreSQL standard-2 (9.5.8)
• Target: AWS RDS PostgreSQL (9.6.3) at sa-east-1 region with rds.logical_replication=1
• DMS: Instance at us-east-1 region

Full Cloudwatch logs:

2017-10-25T17:43:27 [TASK_MANAGER ]I: Task 'WKJYBNDMNIRS3AEA77T5BP5RN4' running full load and CDC in resume mode after recoverable error, retry #3 (replicationtask.c:1078)
2017-10-25T17:43:27 [TASK_MANAGER ]I: Task Id: bb955197-648a-4177-8653-677b5daf5a6e (replicationtask.c:2682)
2017-10-25T17:43:30 [TASK_MANAGER ]I: Creating threads for all components (replicationtask.c:1664)
2017-10-25T17:43:31 [TASK_MANAGER ]I: Threads for all components were created (replicationtask.c:1832)
2017-10-25T17:43:31 [TASK_MANAGER ]I: Task initialization completed successfully (replicationtask.c:2741)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Going to connect to ODBC connection string: PROTOCOL=7.4-0;DRIVER={PostgreSQL};SERVER=XXXXX.rds.amazonaws.com;DATABASE=XXXXX;PORT=5432;sslmode=verify-ca;pqopt=sslrootcert=/rdsdbdata/rds-metadata/certificates/QQSFSDJTBLFXQLQ6VRVDDAYKCM/dms-ca-cert.pem;UID=USERID; (odbc_endpoint_imp.c:2236)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Target endpoint 'PostgreSQL' is using provider syntax 'PostgreSQLLinux' (provider_syntax_manager.c:589)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Transaction size is 10000. Array size is 1000. (odbc_endpoint_imp.c:837)
2017-10-25T17:43:31 [TARGET_APPLY ]I: CSV files will not be kept (odbc_endpoint_imp.c:1519)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Error CSV files will not be kept (odbc_endpoint_imp.c:1528)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Working in transactional apply mode (endpointshell.c:1436)
2017-10-25T17:43:31 [SOURCE_CAPTURE ]I: Going to connect to ODBC connection string: Driver={PostgreSQL Unicode(x64)};Server=SERVER_AT_HEROKU.compute-1.amazonaws.com;Port=5432;Database=XXXX;UseDeclareFetch=1;Fetch=10000;Uid=USERID;Pwd=PASSWORD;sslmode=require (postgres_endpoint_util.c:252)
2017-10-25T17:43:31 [SOURCE_CAPTURE ]I: Source endpoint 'PostgreSQL Source' is using provider syntax '<default>' (provider_syntax_manager.c:583)
2017-10-25T17:43:34 [TASK_MANAGER ]I: All stream components were initialized (replicationtask.c:2547)
2017-10-25T17:43:34 [SOURCE_CAPTURE ]I: Initial position is set to 'now' (postgres_endpoint_util.c:343)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]I: Replication slot 'wkjybndmnirs3aea_00016385_bb955197_648a_4177_8653_677b5daf5a6e' does NOT exist. (postgres_endpoint_capture.c:293)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]I: Initial positioning requested is 'now' (postgres_endpoint_capture.c:313)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Failed to execute statement: 'SELECT xlog_position FROM pg_create_logical_replication_slot('wkjybndmnirs3aea_00016385_bb955197_648a_4177_8653_677b5daf5a6e', 'test_decoding')' [1022502] (ar_odbc_stmt.c:2415)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: RetCode: SQL_ERROR SqlState: 42501 NativeError: 1 Message: ERROR: must be superuser or replication role to use replication slots;, Error while executing the query [1022502] (ar_odbc_stmt.c:2421)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Creating replication slot 'wkjybndmnirs3aea_00016385_bb955197_648a_4177_8653_677b5daf5a6e' failed. [1022502] (postgres_endpoint_capture.c:334)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Error executing source loop [1022502] (streamcomponent.c:1581)
2017-10-25T17:43:35 [TASK_MANAGER ]E: Task error notification received from subtask 0, thread 0 [1022502] (replicationtask.c:2375)
2017-10-25T17:43:35 [TASK_MANAGER ]E: Stream component failed at subtask 0, component st_0_QVUF6NET6KSB26SCIIRH4UQZAQ [1022502] (subtask.c:1350)
2017-10-25T17:43:35 [TASK_MANAGER ]W: Task 'WKJYBNDMNIRS3AEA77T5BP5RN4' encountered a recoverable error (repository.c:4749)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Stream component 'st_0_QVUF6NET6KSB26SCIIRH4UQZAQ' terminated [1022502] (subtask.c:1513)
2017-10-25T17:43:40 [TASK_MANAGER ]I: Subtask #0 ended (replicationtask_util.c:937)
2017-10-25T17:43:40 [TASK_MANAGER ]I: Task management thread terminated (replicationtask.c:3210)

Answer 1

The prereqs for DMS state that the user for the source db needs to have superuser attribute; Heroku doesn't allow this.

https://devcenter.heroku.com/articles/heroku-postgresql

Answer 2

Maybe you can create a lambda within a docker container that has the heroku cli installed and you can add:

heroku pg:backups:capture
heroku pg:backups:download

to the lambda script. You could then upload the zip to s3, and then fire off a script to load the data into RDS