load all databases from the source to ADLG2(azure data lake gen2)

Question

is there a way to load all databases from the source SQL Server to the data lake as it is?

I tried to load each database with his tables but I am asking if there was a way to load all databases as it is to the data lake

Answer 1

One of the ways to load all databases from SQL to ADLG2 is by using Azure Data Factory

Follow the below procedure to load all databases from SQL to ADLG2:

1. First create one pipeline and take script activity in it add linked service with master database select script as Query And give the following query:

    SELECT name, database_id, create_date  
    FROM sys.databases;

2. Then take ForEach activity and int its settings give Items as so it will fetch output of script activity

    @activity('Script1').output.resultSets[0].rows

3. In for each activity take one lookup activity, create and add linked service for database with dynamic values

In that dataset add Db name parameter

Noe send this parameter value to linked service properties as below

Lookup activity settings

SELECT table_Schema, TABLE_NAME, TABLE_CATALOG
FROM information_Schema.tables
WHERE TABLE_TYPE =  'BASE TABLE'sql

4. now take execute pipeline activity click on new in that pipeline create lookupOP parameter with array datatype and in execute pipeline pass the value to it as output of lookup as @activity('Lookup1').output.value

5. In that new pipeline take ForEach activity and passthe parameter we created as items

6. In that for each activity take one copy activity and for the source dataset create linked service on SQL database with dynamic values as we created previously

In this dataset create parameters for database name, table name and schema name

now add these dynamic values to linked service properties and Table name, table schema

Copy activity source setting:

create parameters in sink dataset

now add these dynamic values to folder name, file name

Copy activity Sink settings

Output

creating folder of database name and in that folder loading tables of that particular database