[英]how to improve the performance of bulk transaction in spring mvc
i am working with java spring mvc with JPA and hibernate.我正在使用 java spring mvc 与 JPA 和 ZCB1F008EEFE65012C4EFZA2C3。 If i want to process 2000 transactions, can i process them in batches of 500 transactions using threads putting in consideration that the single transaction should be processed by the same thread?is there any other way to do this?
如果我想处理 2000 个事务,我可以使用线程分批处理 500 个事务,考虑到单个事务应该由同一个线程处理吗?还有其他方法吗?
Hibernate doesn't enable batching by default. Hibernate 默认不启用批处理。 This means that it'll send a separate SQL statement for each insert.
这意味着它将为每个插入发送单独的 SQL 语句。
To insert so massive dataset in once you should use batch insert.要一次性插入如此庞大的数据集,您应该使用批量插入。 Here the hibernate properties:
hibernate.jdbc.batch_size = 50
and hibernate.order_inserts = true
这里的 hibernate 属性:
hibernate.jdbc.batch_size = 50
和hibernate.order_inserts = true
The first property tells Hibernate to collect inserts in batches of 50 (50 is only used here for as example..).第一个属性告诉 Hibernate 以 50 个为一组收集插入(此处仅使用 50 个作为示例......)。 The order_inserts property tells Hibernate to take the time to group inserts by entity, creating larger batches.
order_inserts 属性告诉 Hibernate 花时间按实体对插入进行分组,从而创建更大的批次。
Then you should choose between explicit or implicit flushing.然后您应该在显式或隐式刷新之间进行选择。
flush
and clear
flush
和clear
for (int i = 0; i < 10; i++) {
if (i > 0 && i % BATCH_SIZE == 0) {
entityManager.flush();
entityManager.clear();
}
A a = new A();
entityManager.persist(a);
}
So why using 2nd option?那么为什么要使用第二个选项呢?
When we persist an entity, Hibernate stores it in the persistence context.当我们持久化一个实体时,Hibernate 将它存储在持久化上下文中。 For example, if we persist 2,000 entities in one transaction, we'll end up having 2,000 entity instances in memory, possibly causing an OutOfMemory...
例如,如果我们在一个事务中持久化 2,000 个实体,我们最终将在 memory 中拥有 2,000 个实体实例,这可能会导致 OutOfMemory...
Other problem... as you said you're using MySQL.其他问题......正如你所说,你正在使用 MySQL。 hibernate batch option not work with IdGenerator
IDENTITY
so you can't use batch insert with AUTO-INCREMENT
feature of mysql. hibernate 批处理选项不适用于 IdGenerator
IDENTITY
,因此您不能将批处理插入与 mysql 的AUTO-INCREMENT
功能一起使用。 As sequence generation doesn't exist in MySQL you're stuck with TABLE
generator.由于 MySQL 中不存在序列生成,因此您被
TABLE
生成器困住了。 But table generator is less performant than identity one.但是表生成器的性能不如标识一。 To mention Vlad mihalcea (quote from his famous book
High Performance Java Persistence
)提到 Vlad mihalcea(引自他的名著《
High Performance Java Persistence
》)
When using MySQL and need lot of inserts It is a good idea to use JOOQ framework for that.
当使用 MySQL 并且需要大量插入时,使用 JOOQ 框架是一个好主意。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.