简体   繁体   English

用大量数据填充MS SQL数据库的最快方法是什么

[英]What is the fastest approach to populate MS SQL Database with large amount of data

Dilemma: 困境:

I am about to perform population of data on MS SQL Server (2012 Dev Edition). 我即将在MS SQL Server(2012开发版)上执行数据填充。 Data is based on production data. 数据基于生产数据。 Amount is around 4TB (around 250 million items). 数量约为4TB(约2.5亿件商品)。

Purpose: 目的:

To test performance on full text search and on regular index as well. 测试全文搜索和常规索引的性能。 Target number should be around 300 million items around 500K each. 目标数量应在3亿件左右,每件50万件左右。

Question: 题:

What should I do before to speed up the process or consequences that I should worry about? 在加快我应该担心的过程或后果之前,我应该做什么?

Ex. 例如

  1. Switching off statistics? 关闭统计信息?
  2. Should I do a bulk insert of 1k items per transaction instead of single transaction? 我应该每笔交易批量插入1k项而不是单笔交易吗?
  3. Simple recovery model? 简单的恢复模式?
  4. Log truncation? 日志截断?

Important: 重要:

I will use sample of 2k of production items to create every random item that will be inserted into database. 我将使用2k的生产项目样本来创建将插入数据库的每个随机项目。 I will use near unique samples generated in c#. 我将使用C#中生成的唯一样本。 It will be one table: 这将是一张桌子:

table 
(
    long[id], 
    nvarchar(50)[index], 
    nvarchar(50)[index], 
    int[index], 
    float, 
    nvarchar(50)[index], 
    text[full text search index]
)

Almost invariably, in a situation like this, and I've had several of them, I've used SSIS. 几乎总是在这种情况下,并且我有几种情况,我都使用过SSIS。 SSIS is the fastest way I know to import large amounts of data into a SQL Server database. SSIS是我知道的将大量数据导入SQL Server数据库的最快方法。 You have complete control over batch (transaction size) and it will perform bulk inserting. 您完全控制了批处理(事务大小),它将执行批量插入。 In addition, if you have transformation requirements, SSIS will handle this with ease. 此外,如果您有转换要求,SSIS将轻松解决这一问题。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM