[英]Is it worth to open multiple database connections for sql insertion
I am writing a project related to mass data fetching.我正在编写一个与海量数据获取相关的项目。 Currently I am using .NET Framework 4.8 and the Mysql package to start connection and insert data to the database server.
目前我正在使用 .NET Framework 4.8 和 Mysql 包来启动连接并将数据插入数据库服务器。
I am going to insert around 400, 000 line/second.我将插入大约 400, 000 行/秒。 I am concern that the SQL connection may become the bottleneck of my program.
我担心 SQL 连接可能会成为我程序的瓶颈。 I would like to know if I create a multitthread connection with the sql and insert the data using a consumer queue, would it be faster and is it worth it(pros and cons)?
我想知道如果我用 sql 创建一个多线程连接并使用消费者队列插入数据,它会更快,是否值得(优点和缺点)?
In my intuitive thought it would be faster but I am not sure how much performance it can provide with respect of the overhead for threads.在我的直觉中,它会更快,但我不确定它在线程开销方面可以提供多少性能。 I am not a SQL expert so it would be nice if someone could explain the pros and cons of opening multiple connections to a SQL on multiple threads.
我不是 SQL 专家,所以如果有人能解释在多个线程上打开多个 SQL 连接的优缺点,那就太好了。
Rumors, opinions, hearsay, facts, version-dependent benchmarks, some personal experience, etc...谣言、观点、传闻、事实、版本相关的基准、一些个人经验等……
Multiple threads will improve throughput, but with limits:多线程将提高吞吐量,但有限制:
Batching:批处理:
LOAD DATA
is the fastest way to INSERT
lots of rows from a single thread at a single time. LOAD DATA
是一次从单个线程INSERT
大量行的最快方法。 But if you include the cost of writing the file to LOAD
, that may make it effectively slower than batched inserting.LOAD
的成本,这可能会使其实际上比批量插入慢。INSERT
is a close second.INSERT
紧随其后。 But it caps out at "hundreds" of rows, when it hits either some limit or "diminishing returns".INSERT
query.INSERT
查询插入一行的 10 倍。 So it (or LOAD DATA
) is worth using for high speed ingestion.LOAD DATA
)值得用于高速摄取。 (Source: many different timed tests.) Source of data:数据来源:
What happens after loading the data?加载数据后会发生什么? Surely this is not a write-only-never-read table.
当然,这不是一个只写永不读的表。
PARTITIONing
is rarely useful, except for eventual purging of old data. PARTITIONing
很少有用,除了最终清除旧数据。 See Partition
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.