如何不断地将数据从本地 SQL Db 迁移到 Azure SQL Db

Question

As a part of Azure Machine Learning process, I need to continually migrate data from on-premises SQL Db to Azure SQL Db using Data Management Gateway .

This Azure official article describes how to: Move data from an on-premises SQL server to SQL Azure with Azure Data Factory . But the details are a bit confusing to me. If someone to briefly describe the process, how would you do that. What are 2-3 main steps one needs to perform on on-premises and 2-3 steps on Azure Cloud ? No details are needed. Note : The solution has to involve using Data Management Gateway

Answer 1

Based on Azure documentation you can use "slices". You can perform a "delta" fetch using a timestamp column as mentioned by this article or using a sequential integer column. To avoid issues about rows not been included on the a slice due to the on-premise server having system date a little behind than Azure system date, is better to use a sequential integer. Below the Input dataset shows how to define slices:

{
  "name": "AzureBlobInput",
  "properties": {
    "type": "AzureBlob",
    "linkedServiceName": "StorageLinkedService",
    "typeProperties": {
      "folderPath": "mycontainer/myfolder/{Year}/{Month}/{Day}/",
      "partitionedBy": [
        { "name": "Year", "value": {"type": "DateTime","date": "SliceStart","format": "yyyy"}},
        { "name": "Month","value": {"type": "DateTime","date": "SliceStart","format": "MM"}},
        { "name": "Day","value": {"type": "DateTime","date": "SliceStart","format": "dd"}}
      ],
      "format": {
        "type": "TextFormat"
      }
    },
    "external": true,
    "availability": {
      "frequency": "Hour",
      "interval": 1
    }
  }
}

You can create an activity and use the availability section to specify a schedule for the activity. You can specify "frequency" (minute, hour, day, etc.) and "interval".

            "scheduler": {
                "frequency": "Hour",
                "interval": 1
            }

Each unit of data consumed or produced by an activity run is called a data slice. The following diagram shows an example of an activity with one input dataset and one output dataset:

The diagram shows the hourly data slices for the input and output dataset. The diagram shows three input slices that are ready for processing. The 10-11 AM activity is in progress, producing the 10-11 AM output slice.

You can access the time interval associated with the current slice in the dataset JSON by using variables: SliceStart and SliceEnd. You can use these variables in your activity JSON to select data from input dataset representing time series data (for example: 8 AM to 9 AM).

You can also set the start date for the pipeline in the past as shown here . When you do so, Data Factory automatically calculates (back fills) all data slices in the past and begins processing them.