Spring Boot - H2 DB import 4GB data.sql OutOfMemory Error

Question

I want to import my data in data.sql that is about 4GB But OutOfMemory - heap space error occured.

I already increase my heap space in IntelliJ but is still not working. (-Xms16g -Xmx16g)

My error is:

Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3332) ~[na:1.8.0_211]
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) ~[na:1.8.0_211]
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) ~[na:1.8.0_211]
    at java.lang.StringBuilder.append(StringBuilder.java:136) ~[na:1.8.0_211]
    at org.springframework.jdbc.datasource.init.ScriptUtils.readScript(ScriptUtils.java:388) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.jdbc.datasource.init.ScriptUtils.readScript(ScriptUtils.java:330) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.jdbc.datasource.init.ScriptUtils.executeSqlScript(ScriptUtils.java:582) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.jdbc.datasource.init.ResourceDatabasePopulator.populate(ResourceDatabasePopulator.java:254) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.jdbc.datasource.init.DatabasePopulatorUtils.execute(DatabasePopulatorUtils.java:49) ~[spring-jdbc-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializer.runScripts(DataSourceInitializer.java:202) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializer.initSchema(DataSourceInitializer.java:119) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializerInvoker.onApplicationEvent(DataSourceInitializerInvoker.java:91) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.boot.autoconfigure.jdbc.DataSourceInitializerInvoker.onApplicationEvent(DataSourceInitializerInvoker.java:38) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:403) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:360) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.boot.autoconfigure.orm.jpa.DataSourceInitializedPublisher.publishEventIfRequired(DataSourceInitializedPublisher.java:99) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.boot.autoconfigure.orm.jpa.DataSourceInitializedPublisher.postProcessAfterInitialization(DataSourceInitializedPublisher.java:90) ~[spring-boot-autoconfigure-2.2.4.RELEASE.jar:2.2.4.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:431) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda$151/1711185459.getObject(Unknown Source) ~[na:na]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1108) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:868) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]

And my applicaiton.yml(h2 config) is:

spring:
  h2:
    console:
      enabled: true
      path: /h2-console

  jpa:
    database: h2
    showSql: false
    hibernate:
      ddl-auto: create
    properties:
      hibernate.dialect: org.hibernate.dialect.H2Dialect
      hibernate.format_sql: true
      hibernate.use_sql_comments: true

  datasource:
    platform: h2
    url: jdbc:h2:mem:testdb;MODE=MYSQL
    username: sa
    password:
    initialization-mode: embedded

How can I solve this error?

Answer 1

转到终端并使用此命令启动您的应用程序

java -jar your_app_name.jar -Xms16g -Xmx16g

Answer 2

tl;dr : the spring database sql initialisation wasn't meant to be used with large files

OutOfMemory is coming from your usage of the spring database sql initialisation

https://docs.spring.io/spring-boot/docs/current/reference/html/howto.html#howto-initialize-a-database-using-spring-jdbc

It wasn't meant to execute such a large file.

Spring will first load the file in memory During the loading the file is added line by line to a StringBuilder that get constantly re-allocated leading to temporary spike in memory allocation

See https://github.com/spring-projects/spring-framework/blob/0ddc62e87f9e0c9fb664ba085bcdf7ca61e697dd/spring-jdbc/src/main/java/org/springframework/jdbc/datasource/init/ScriptUtils.java#L380-L392

Depending of your jvm version the memory needed to store the file in memory can be a lot more than the original file size (as it may be stored in unicode)

Then the resulting StringBuilder is converted to a String (temporary memory allocation)

The string is then splitted by sql statement all added to an arraylist (doubling the memory needed by your script)

See https://github.com/spring-projects/spring-framework/blob/0ddc62e87f9e0c9fb664ba085bcdf7ca61e697dd/spring-jdbc/src/main/java/org/springframework/jdbc/datasource/init/ScriptUtils.java#L596