Moving to Azure Synapse from Azure SQL Database

Question

We are currently ingesting data from application databases into an Azure SQL Database. The size is around 600 GB now (which mainly distributed to only 3 fact tables, the rest of the tables are master data which is quite small) and it's running on 40 vCores (we use it a lot for reporting, so need a high number of vCores).

Some difficulties I'm currently facing:

1. Data copy from source to sink usually takes a really long time. The approach we are using is to delete all records for this month, and then copy this month's data from application db over. Writing to sink usually takes a lot of time as well (due to the indices on the fact table I believe).

2. High data I/O whenever someone pulls a big query.

Here to hope someone can shed some lights on how to make the setup works faster.

Thanks!

Answer 1

Though i needed some more info , but i think i can share some pointers . When you mentioned big query taking more time " , have you checked the query and make sure that the indices are there on the required columns and they are updated regulary ? It appears that you have 3 tables with lot of data , is the query are slow on all three ? ( If I were you i will try the divide the problems into smaller issues and investigate each one of them )

On the copy part , before you copy you will have to select the data and so we will have to improve the query perofiormance which I mentioned above . How are you copying data ? Since I see ADF tag , I am assuming its ADF . Are you copying data sequentially ? I mean copy data to BigTable1 then copy BigTable2 then BigTable 3 ? You can explore the possiblilty of copying data in parallel . I am not sure how have you implmeneted the logic in ADF but three copy avtivity one below the other will do the trick .

In each copy activity you have the option to set parallelism and also the batchcount , I could suggest to take a look on that .

Performanace problem is very difficult to help with unless you have access to the data :) let me know how it goes .

https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-performance-features#parallel-copy

Thanks

Himanshu