简体繁体 English

从 Azure SQL 数据库迁移到 Azure Synapse

[英]Moving to Azure Synapse from Azure SQL Database

原文 2020-08-25 06:44:46 2 1 azure/ azure-sql-database/ azure-data-factory/ azure-synapse

We are currently ingesting data from application databases into an Azure SQL Database.我们目前正在将应用程序数据库中的数据引入 Azure SQL 数据库。 The size is around 600 GB now (which mainly distributed to only 3 fact tables, the rest of the tables are master data which is quite small) and it's running on 40 vCores (we use it a lot for reporting, so need a high number of vCores).现在的大小大约是 600 GB（主要分布到只有 3 个事实表，其余的表是主数据，非常小）并且它在 40 个 vCore 上运行（我们经常使用它来报告，所以需要很大的数字） vCore 数量）。

Some difficulties I'm currently facing:我目前面临的一些困难：

Data copy from source to sink usually takes a really long time.从源到接收器的数据复制通常需要很长时间。 The approach we are using is to delete all records for this month, and then copy this month's data from application db over.我们使用的方法是delete本月的所有记录，然后copy本月的数据从应用程序数据库中copy过来。 Writing to sink usually takes a lot of time as well (due to the indices on the fact table I believe).写入 sink 通常也需要很多时间（由于我相信事实表上的索引）。
High data I/O whenever someone pulls a big query.每当有人提出大查询时，数据 I/O 就很高。

Here to hope someone can shed some lights on how to make the setup works faster.在这里希望有人可以阐明如何使设置工作得更快。

Thanks!谢谢！

1 个解决方案

Though i needed some more info , but i think i can share some pointers .虽然我需要更多信息，但我想我可以分享一些建议。 When you mentioned big query taking more time " , have you checked the query and make sure that the indices are there on the required columns and they are updated regulary ? It appears that you have 3 tables with lot of data , is the query are slow on all three ? ( If I were you i will try the divide the problems into smaller issues and investigate each one of them )当您提到需要更多时间的大查询时，您是否检查过查询并确保索引位于所需的列上并且它们定期更新？看起来您有 3 个包含大量数据的表，查询是不是很慢？在所有三个方面？（如果我是你，我会尝试将问题分成更小的问题并调查每个问题）

On the copy part , before you copy you will have to select the data and so we will have to improve the query perofiormance which I mentioned above .在复制部分，在复制之前，您必须选择数据，因此我们必须改进我上面提到的查询性能。 How are you copying data ?你是如何复制数据的？ Since I see ADF tag , I am assuming its ADF .因为我看到 ADF 标签，所以我假设它的 ADF 。 Are you copying data sequentially ?您是否按顺序复制数据？ I mean copy data to BigTable1 then copy BigTable2 then BigTable 3 ?我的意思是将数据复制到 BigTable1 然后复制 BigTable2 然后 BigTable 3 ？ You can explore the possiblilty of copying data in parallel .您可以探索并行复制数据的可能性。 I am not sure how have you implmeneted the logic in ADF but three copy avtivity one below the other will do the trick .我不确定你是如何在 ADF 中实现逻辑的，但是三个副本 avtivity 一个在另一个下面会起作用。

In each copy activity you have the option to set parallelism and also the batchcount , I could suggest to take a look on that .在每个复制活动中，您都可以选择设置并行度和批次计数，我建议您查看一下。

Performanace problem is very difficult to help with unless you have access to the data :) let me know how it goes .除非您有权访问数据，否则很难解决性能问题:) 让我知道它是怎么回事。

https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-performance-features#parallel-copy https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-performance-features#parallel-copy

Thanks谢谢

Himanshu喜满树