PostgreSQL 使用 AWS DMS 迁移后主键序列丢失

Question

I recently migrated a self hosted Postgres database to AWS RDS using the AWS Migration Service.
Postgres Version: 10.6

I have noticed that all of my primary keys are no longer set to "sequence", and when I attempt to manually add a sequence it starts at 1 instead of continuing the count that is already set.

I use Rails with the database, so my sql skills are pretty low. I can generally find my way around inserts and updates, but this is not a realm I have much experience in.

My question has 2 parts:

1. How do I fix a single table? I want to understand and test what I am doing before moving on.
2. Is there a way I can apply this fix to every table I have without needing to manually modify each table?

Answer 1

After @a_horse_with_no_name pointed me in the right direction and chatting with AWS I am able to answer my own question, at least if you are using AWS Database Migration Service (DMS).

The problem is, DMS only focuses on the data itself and not really the schema (which to me seems like a major oversight, especially if your using the same database technology but that is another issue). So the schema itself is not migrated. The documentation does not really make this clear.

To fix this issue:

1. Stop (if it still exists) the existing AWS DMS migration
2. Drop the existing migrated database, and create a new empty schema to use
3. Follow the steps here https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Installing.html to install and setup the Amazon Schema Conversation Tool (SCT)
4. Once you are connected to both databases, follow the steps here https://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/CHAP_Converting.html to "convert" your schema (I did the entire "public" schema for this database to ensure everything is covered
5. Create or modify your AWS DMS Migration, ensuring Target Table Preparation Mode = "TRUNCATE" and disable foreign keys on the target database. If modifying, make sure when asked you "RESTART" not resume

What I have not yet tested is how to handle the fact that I am migrating a live database. So the sequences may be out of date on the target database when the migration is done. I believe I can just later go into SCT and only migrate the sequences but I have not tested this yet.

Answer 2

Unless you missed something and the migration could be processed once more (with different parameters? I have never used AWS Migration Service but I presume it should preserve the serial -iness of your columns...) you'll need to re-create the sequences too.

---

I encountered a similar situation a year or so ago, and wrote this answer on SO .

Here's the gist of it:

CREATE SEQUENCE foo_a_seq OWNED BY foo.a;
SELECT setval('foo_a_seq', coalesce(max(a), 0)) FROM foo;
ALTER TABLE foo ALTER COLUMN a SET DEFAULT nextval('foo_a_seq'); 

This would create a sequence foo_a_seq , with its nextval being one higher than the max value in foo.a (or 1 if there is no foo record).

I also went ahead and set up a Function to quickly apply to all the tables/columns in need:

CREATE OR REPLACE FUNCTION make_into_serial(table_name TEXT, column_name TEXT) RETURNS INTEGER AS $$
DECLARE
    start_with INTEGER;
    sequence_name TEXT;
BEGIN
    sequence_name := table_name || '_' || column_name || '_seq';
    EXECUTE 'SELECT coalesce(max(' || column_name || '), 0) + 1 FROM ' || table_name
            INTO start_with;
    EXECUTE 'CREATE SEQUENCE ' || sequence_name ||
            ' START WITH ' || start_with ||
            ' OWNED BY ' || table_name || '.' || column_name;
    EXECUTE 'ALTER TABLE ' || table_name || ' ALTER COLUMN ' || column_name ||
            ' SET DEFAULT nextVal(''' || sequence_name || ''')';
    RETURN start_with;
END;
$$ LANGUAGE plpgsql VOLATILE;

Use it like so:

SELECT make_into_serial('foo', 'a');

Answer 3

Sequence, Index, and Constraint are not migrated and it is mentioned in the official docs on AWS.

You can use this source . This will help you to migrate Sequence, Index, and Constraint at once.

Answer 4

For any folks using macOS and aren't able to use the SchemaConversionTool suggested in one of the answers above, follow the steps below:

1. Empty the schema in the target database in order to start fresh

   DROP SCHEMA public;

2. Generate a SQL script which contains all the DDL statements from the source database's schema. If you use dbeaver then right click on schema name and click on Generate SQL

3. Run the commands in the generated SQL script from step 2 in the target database so that the schemas are now in sync with each other

4. Go to AWS DMS and start the task with target table preparation mode set to truncate