使用 AWS CDK 創建 AWS DMS 任務

Question

我正在嘗試使用 AWS CDK 創建 AWS DMS 任務。 但我不知道從哪里開始。 我找不到關於如何使用 CDK 創建 DMS 任務的好文檔。 我找到了關於這兩個主題的文章，但找不到解決這個問題的文章 - 討論如何使用 CDK 創建 DMS 任務。

任何人都可以指出我正確的文章來解釋這個或幫助我這樣做嗎？

PS - 我已經用 dms maven 依賴項初始化了項目。 我正在使用 JAVA。

謝謝

Answer 1

沒有 CDK 結構來簡化 DMS 的工作。 因此，您必須使用 CloudFormation 資源：CfnEndpoint、CfnReplicationTask 等。

我提供了以下示例來幫助您入門，但請注意，DMS CloudFormation 資源非常具有挑戰性。

import * as cdk from '@aws-cdk/core';
import * as dms from '@aws-cdk/aws-dms';

export class DmsStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create a subnet group that allows DMS to access your data
    const subnet = new dms.CfnReplicationSubnetGroup(this, 'SubnetGroup', {
      replicationSubnetGroupIdentifier: 'cdk-subnetgroup',
      replicationSubnetGroupDescription: 'Subnets that have access to my data source and target.',
      subnetIds: [ 'subnet-123', 'subnet-456' ],
    });

    // Launch an instance in the subnet group
    const instance = new dms.CfnReplicationInstance(this, 'Instance', {
      replicationInstanceIdentifier: 'cdk-instance',

      // Use the appropriate instance class: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.Types.html
      replicationInstanceClass: 'dms.t2.small',

      // Setup networking
      replicationSubnetGroupIdentifier: subnet.replicationSubnetGroupIdentifier,
      vpcSecurityGroupIds: [ 'sg-123' ],
    });

    // Create endpoints for your data, see https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dms-endpoint.html
    const source = new dms.CfnEndpoint(this, 'Source', {
      endpointIdentifier: 'cdk-source',
      endpointType: 'source',
      engineName: 'mysql',

      serverName: 'source.database.com',
      port: 3306,
      databaseName: 'database',
      username: 'dms-user',
      password: 'password-from-secret',
    });

    const target = new dms.CfnEndpoint(this, 'Target', {
      endpointIdentifier: 'cdk-target',
      endpointType: 'target',
      engineName: 's3',

      s3Settings: {
        bucketName: 'target-bucket'
      },
    });

    // Define the replication task
    const task = new dms.CfnReplicationTask(this, 'Task', {
      replicationInstanceArn: instance.ref,

      migrationType: 'full-load', // https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dms-replicationtask.html#cfn-dms-replicationtask-migrationtype
      sourceEndpointArn: source.ref,
      targetEndpointArn: target.ref,
      tableMappings: JSON.stringify({
        "rules": [{
          "rule-type": "selection",
          "rule-id": "1",
          "rule-name": "1",
          "object-locator": {
            "schema-name": "%",
            "table-name": "%"
          },
          "rule-action": "include"
        }]
      })
    })
  }
}

Answer 2

只是對先前設置的補充——由於 DMS 上的一些變化——它不會等到 IAM 資源被創建——所以將其添加為對 su.netgroup 資源的依賴並為 su.netg 添加對實例的依賴，這應該可以節省你有 2-3 個小時為什么它不工作但在筒倉的代碼中工作....

import * as cdk from '@aws-cdk/core';
import * as dms from '@aws-cdk/aws-dms';
import { 
  ManagedPolicy, 
  Role, 
  ServicePrincipal, 
  PolicyStatement, 
  Effect 
} from '@aws-cdk/aws-iam';

import { App, Construct, Stack } from "@aws-cdk/core";

const app = new App();

app.synth()



export class DmsStack extends cdk.Stack {
  role: Role;
  public constructor(scope:cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    
    const dmsVPCServiceRole = new Role(this, 'dms-vpc-role', {
      assumedBy: new ServicePrincipal('dms.amazonaws.com'),
      roleName: 'dms-vpc-role'
    });

    // Add a policy to a Role
    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['*'],
        actions: [            
          'sts:AssumeRole',
        ]
      })
    );

    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['*'],
        actions: [            
          'dms:*',
        ]
      })
    );

    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['*'],
        actions: [            
          "kms:ListAliases", 
          "kms:DescribeKey"
        ]
      })
    );

    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['*'],
        actions: [            
          "iam:GetRole",
          "iam:PassRole",
          "iam:CreateRole",
          "iam:AttachRolePolicy"
        ]
      })
    );

    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['*'],
        actions: [       
          "ec2:CreateVpc", 
          "ec2:CreateSubnet",      
          "ec2:DescribeVpcs",
          "ec2:DescribeInternetGateways",
          "ec2:DescribeAvailabilityZones",
          "ec2:DescribeSubnets",
          "ec2:DescribeSecurityGroups",
          "ec2:ModifyNetworkInterfaceAttribute",
          "ec2:CreateNetworkInterface",
          "ec2:DeleteNetworkInterface"
        ]
      })
    );


    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['*'],
        actions: [            
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams",
          "logs:FilterLogEvents",
          "logs:GetLogEvents"
        ]
      })
    );

    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['arn:aws:s3:::BUCKETNAME/*'],
        actions: [            
          "s3:PutObject",
          "s3:DeleteObject",
          "s3:PutObjectTagging"
        ]
      })
    );

    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['arn:aws:s3:::BUCKETNAME'],
        actions: [            
          "s3:ListBucket"
        ]
      })
    );

    dmsVPCServiceRole.addToPolicy(
      new PolicyStatement({
        effect: Effect.ALLOW,
        resources: ['arn:aws:s3:::BUCKETNAME'],
        actions: [            
          "s3:GetBucketLocation"
        ]
      })
    );

    const dmsVpcManagementRolePolicy = ManagedPolicy.fromManagedPolicyArn(
      this, 
      'AmazonDMSVPCManagementRole', 
      'arn:aws:iam::aws:policy/service-role/AmazonDMSVPCManagementRole'
    );

    dmsVPCServiceRole.addManagedPolicy(dmsVpcManagementRolePolicy);





    
    // // Create a subnet group that allows DMS to access your data
    const subnet = new dms.CfnReplicationSubnetGroup(this, 'SubnetGroup', {
      replicationSubnetGroupIdentifier: 'cdk-subnetgroup',
      replicationSubnetGroupDescription: 'Subnets that have access to my data source and target.',
      subnetIds: ['subnet-01', 'subnet-02']
    });

    subnet.node.addDependency(dmsVPCServiceRole);



    const instance = new dms.CfnReplicationInstance(this, 'Instance', {
      replicationInstanceIdentifier: 'cdk-instance',

      // Use the appropriate instance class: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_ReplicationInstance.Types.html
      replicationInstanceClass: 'dms.t2.small',

      // Setup networking
      replicationSubnetGroupIdentifier: subnet.replicationSubnetGroupIdentifier,
      vpcSecurityGroupIds: [ 'sg-041c1c796c1130121' ],
    });

    instance.node.addDependency(subnet)


  }
}

Answer 3

您可以創建一個 AWS CDK 任務來捕獲來自源數據存儲的持續更改。 您可以在遷移數據時執行此捕獲。 您還可以創建一個任務，在完成到支持的目標數據存儲的初始（完全加載）遷移后捕獲正在進行的更改。 此過程稱為持續復制或更改數據捕獲 (CDC)。 AWS CDK 在從源數據存儲復制正在進行的更改時使用此過程。 此過程通過使用數據庫引擎的本機 API 收集對數據庫日志的更改來工作。