Spring Boot - H2 DB 导入 4GB data.sql OutOfMemory 错误

Question

我想在大约 4GB 的 data.sql 中导入我的数据但是 OutOfMemory - 发生堆空间错误。

我已经在 IntelliJ 中增加了堆空间，但仍然无法正常工作。 (-Xms16g -Xmx16g)

我的错误是：

Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_211]
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) ~[na:1.8.0_211]
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) ~[na:1.8.0_211]
    at java.lang.StringBuilder.append(StringBuilder.java:136) ~[na:1.8.0_211]
    at org.springframework.jdbc.datasource.init.ScriptUtils.readScript(ScriptUtils.java:388) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.jdbc.datasource.init.ScriptUtils.readScript(ScriptUtils.java:330) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.jdbc.datasource.init.ScriptUtils.executeSqlScript(ScriptUtils.java:582) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.jdbc.datasource.init.ResourceDatabasePopulator.populate(ResourceDatabasePopulator.java:254) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.jdbc.datasource.init.DatabasePopulatorUtils.execute(DatabasePopulatorUtils.java:49) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializer.runScripts(DataSourceInitializer.java:202) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializer.initSchema(DataSourceInitializer.java:119) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializerInvoker.onApplicationEvent(DataSourceInitializerInvoker.java:91) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializerInvoker.onApplicationEvent(DataSourceInitializerInvoker.java:38) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:403) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:360) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.boot.autoconfigure.orm.jpa.DataSourceInitializedPublisher.publishEventIfRequired(DataSourceInitializedPublisher.java:99) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.boot.autoconfigure.orm.jpa.DataSourceInitializedPublisher.postProcessAfterInitialization(DataSourceInitializedPublisher.java:90) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:431) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$151/1711185459.getObject(Unknown Source) ~[na:na]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1108) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:868) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]

我的 applicaiton.yml(h2 config) 是：

spring:
  h2:
    console:
      enabled: true
      path: /h2-console

  jpa:
    database: h2
    showSql: false
    hibernate:
      ddl-auto: create
    properties:
      hibernate.dialect: org.hibernate.dialect.H2Dialect
      hibernate.format_sql: true
      hibernate.use_sql_comments: true

  datasource:
    platform: h2
    url: jdbc:h2:mem:testdb;MODE=MYSQL
    username: sa
    password:
    initialization-mode: embedded

我该如何解决这个错误？

Answer 1

转到终端并使用此命令启动您的应用程序

java -jar your_app_name.jar -Xms16g -Xmx16g

Answer 2

tl;dr：spring 数据库 sql 初始化不适合用于大文件

OutOfMemory 来自您对 spring 数据库 sql 初始化的使用

https://docs.spring.io/spring-boot/docs/current/reference/html/howto.html#howto-initialize-a-database-using-spring-jdbc

它并不是要执行这么大的文件。

Spring 将首先在内存中加载文件在加载过程中，文件被逐行添加到 StringBuilder 中，该 StringBuilder 不断重新分配，导致内存分配出现临时峰值

参见https://github.com/spring-projects/spring-framework/blob/0ddc62e87f9e0c9fb664ba085bcdf7ca61e697dd/spring-jdbc/src/main/java/org/springframework/jdbc/datasource/init/ScriptUtils.java#L3280-L39

根据您的 jvm 版本，将文件存储在内存中所需的内存可能比原始文件大小大得多（因为它可能以 unicode 格式存储）

然后将生成的 StringBuilder 转换为 String（临时内存分配）

然后该字符串由 sql 语句拆分，全部添加到数组列表中（将脚本所需的内存加倍）

参见https://github.com/spring-projects/spring-framework/blob/0ddc62e87f9e0c9fb664ba085bcdf7ca61e697dd/spring-jdbc/src/main/java/org/springframework/jdbc/datasource/init/ScriptUtils.java#L596