Moving relational data from MSSQL (Azure SQL Server) to MySQL database

Question

What would be the correct way to copy a large dataset from MSSQL (Azure SQL Server) to self hosted MySQL database? To add some complexity this needs to be done every night to update prices, products etc. which a web app uses.

At the moment we have a Node.js script which queries everything table by table and creates the rows with the same data, foreign and primary keys in the MySQL database. This works fine with the smaller tables. The main issue is with Customers and Prices tables, and many-to-many relations between them. At the moment there is CustomersPrices table in MSSQL with the defined relations and it has over 44 million rows. Customers table by itself has only 3000 rows and Prices has only 0,5 million.

Down below is the schema of Customers, Prices and CustomersPrices from MSSQL:

model CustomersPrices {
  PriceID       Int?
  CustomerID    Int?
  Private_Label String?
  Customers     Customers? @relation(fields: [CustomerID], references: [ID])
  Prices        Prices?    @relation(fields: [PriceID], references: [ID])
}

model Customers {
  ID                      Int               @id @default(autoincrement())
  No_                     String?
  Name                    String?
  Customer_Price_Group    String?
  Customer_Disc__Group    String?
  Customer_PL_Price_Group String?
  Customer_PL_Disc__Group String?
  PL_Text                 String?
  Closed_Status           String?
  Credit_Limit__LCY_      Decimal?
  Current_Balance__LCY_   Decimal?
  Remaining_Limit__LCY_   Decimal?
  Salesperson_Code        String?
  Responsibility_Center   String?
  Address                 String?
  Address_2               String?
  Post_Code               String?
  City                    String?
  Shipment_Method_Code    String?
  Bill_to_Customer_No_    String?
  Show_Limit_Info         Int?
  Primary_Contact_No_     String?
  InsertedDatetime        DateTime
  SalesPersonID           Int?
  Salesperson             Salesperson?      @relation(fields: [SalesPersonID], references: [ID])
  Contact                 Contact[]
  CustomersPrices         CustomersPrices[]
  Sales_Header            Sales_Header[]
}

model Prices {
  ID                        Int               @id @default(autoincrement())
  Item_No_                  String?
  Variant_Code              String?
  Sales_Type                String?
  Sales_Code                String?
  Unit_of_Measure_Code      String?
  Sales_Minimum_Quantity    Decimal?
  Unit_Price                Decimal?
  Discount_Minimum_Quantity Decimal?
  Discount_Pct              Decimal?
  Starting_Date             DateTime?
  Ending_Date               DateTime?
  InsertedDatetime          DateTime 
  ProductID                 Int?
  Products                  Products?         @relation(fields: [ProductID], references: [ID])
  CustomersPrices           CustomersPrices[]
}

As we have been instructed we are not able to use the MSSQL for the web app directly.

At the moment we are trying to use Node.js with Prisma.io (to work with MySQL) and Tedious (to query MSSQL).

Feeling really stuck at the moment. How would you build this? Thank you in advance.

Answer 1

You don't want to do this pragmatically. There are tools to address data migrations. Check out how you can use MySQL Workbench to accomplish this: https://www.mysql.com/products/workbench/migrate/