Raspberry PI 2 Java 8 JVM错误

Question

I am in the process of building a web app based on Java 1.8, spring, hibernate, mysql and using tomcat as web server. Right now my app is logging (@5 sec interval) the raspberry's system data like; cpu temperature, cpu core voltage, used memory, free memory, memory buffers, etc. The problem is that from time to time the app is crashing because it's running out of memory and I can't identify the reason.

First, on tomcat I had the following lines in tomcat's startup service :

Environment='CATALINA_OPTS=-Xms256M -Xmx512M -server -XX:+UseParallelGC'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'

and it was crashing in about 8 hours after starting tomcat. I removed those lines and now it is crashing at about 2 days because it is out of memory. Today it was the first time when it crashed without those lines in starting script and my last line inserted in db was:

# id, cpuTemperature, cpuCoreVoltage, usedMemory, freeMemory, memoryBuffers, sdramCVoltage, sdramIVoltage, sdramPVoltage, createdAt, modifiedAt, createdBy, modifiedBy
'52414', '39',          '1.3125',    '958320640', '12447744', '92495872',        '1.2',         '1.2',       '1.225',     '2015-11-23 15:53:58', NULL, NULL, NULL

For this info I am using SystemInfo class from pi4j library. getMemoryUsed() is returning used memory in bits so after a simple math we obtain a used memory of 913.92 Mb.

When app start, it is consuming only 250 Mb ram and after 3 day is crashing the app beacause it's reaching the maximum memory of the raspberry pi.

Spring App Config:

@Configuration
@EnableWebMvc
@EnableScheduling
@ComponentScan({"ro.i2cWebAppController.app.controller","ro.i2cWebAppController.app.serv","ro.i2cWebAppController.app.schedule"})
@Import({SpringSecurityConfig.class})
@EnableJpaRepositories({"ro.i2cWebAppController.app.repo"})
public class AppConfig extends WebMvcConfigurerAdapter {

    private static final Logger logger = Logger.getLogger(AppConfig.class);

    @Override
    public void addResourceHandlers(ResourceHandlerRegistry registry) {
        registry.addResourceHandler("/res/**").addResourceLocations("/WEB-RES/");
        registry.addResourceHandler("/**/**.ico").addResourceLocations("/WEB-RES/img/favicon.ico");
    }

    @Override
    public void configureDefaultServletHandling(DefaultServletHandlerConfigurer configurer) {
        configurer.enable();
    }

    @Bean
    public ViewResolver viewResolver(ContentNegotiationManager manager) {
        InternalResourceViewResolver resolver = new InternalResourceViewResolver();
        resolver.setPrefix("views/");
        resolver.setSuffix(".jsp");
        return resolver;
    }

    @Bean(name = "entityManagerFactory")
    public LocalContainerEntityManagerFactoryBean sessionFactory() {
        LocalContainerEntityManagerFactoryBean lcemfb = new LocalContainerEntityManagerFactoryBean();
        HibernateJpaVendorAdapter hjva = new HibernateJpaVendorAdapter();
        hjva.setShowSql(true);
        //persistence unit
        lcemfb.setDataSource(dataSource());
        lcemfb.setJpaVendorAdapter(hjva);
        lcemfb.setPackagesToScan("ro.i2cWebAppController.app.model");
        lcemfb.setJpaPropertyMap(hibernateProperties());
        return lcemfb;
    }

    @Bean
    public DataSource dataSource() {
        BasicDataSource dataSource = new BasicDataSource();
        dataSource.setDriverClassName("com.mysql.jdbc.Driver");
        if(isUnix()){
            dataSource.setUrl("jdbc:mysql://localhost:3306/i2cwac?autoReconnect=true");
        }else{
            dataSource.setUrl("jdbc:mysql://192.168.1.30:3306/i2cwac?autoReconnect=true");
        }
        dataSource.setUsername("i2cwac");
        dataSource.setPassword("Stefan1234");
        dataSource.setInitialSize(5);
        dataSource.setMaxTotal(20);
        dataSource.setMaxIdle(10);
        dataSource.setMinIdle(5);
        dataSource.setMaxWaitMillis(2000);
        return dataSource;
    }

    @Bean
    @Autowired
    public JpaTransactionManager transactionManager(EntityManagerFactory lcemfb) {
        JpaTransactionManager txManager = new JpaTransactionManager();
        txManager.setEntityManagerFactory(lcemfb);

        return txManager;
    }

    @Bean
    public PersistenceExceptionTranslationPostProcessor exceptionTranslation() {
        return new PersistenceExceptionTranslationPostProcessor();
    }

    Map<String,String> hibernateProperties() {
        Map<String,String> propMap = new HashMap<>();
        propMap.put("hibernate.hbm2ddl.auto", "validate");
        propMap.put("hibernate.dialect", "org.hibernate.dialect.MySQL5InnoDBDialect");
        propMap.put("hibernate.globally_quoted_identifiers", "true");
        propMap.put("hibernate.show_sql", "true");
        return propMap;
    }

    @Bean
    public Integer ambientTemperature(){
        return new Integer(0);
    }

    @Bean
    public SystemInformationService systemInformationService(){
        return new SystemInformationServiceImpl();
    }

    @Bean
    public Boolean isUnix(){
        if(System.getProperty("os.name").toLowerCase().indexOf("win")>=0){
            logger.info("isUnix : false because : " + System.getProperty("os.name"));
            return false;
        }else{
            logger.info("isUnix : true because : " + System.getProperty("os.name"));
            return true;
        }
    }

    @Bean
    public I2CBus i2CBus(){
        I2CBus i2CBus = null;
        if (isUnix()) {
            int count = 0;
            int maxTries = 10;
            while (true) {
                try {
                    i2CBus = I2CFactory.getInstance(I2CBus.BUS_1);
                    break;
                } catch (IOException e) {
                    ++count;
                    logger.error("I2CBUS try " + count);
                    if (count == maxTries) e.printStackTrace();
                }
            }
        }
        if (i2CBus == null){
            logger.info("I2CBUS is NULL");
        }else{
            logger.info("Got I2CBUS " + i2CBus.toString());
        }

        return i2CBus;
    }
}

Scheduler class:

@Component
public class RaspberryPiScheduler {

    @Autowired
    SystemInformationService systemInformationService;

    @Autowired
    RaspberryPiDataService raspberryPiDataService;

    @Autowired
    Boolean isUnix;

    private static final Logger logger = Logger.getLogger(RaspberryPiScheduler.class);

    /**
     * Scheduled method to write data from Raspberry Pi sensors
     */
    @Scheduled(fixedDelay = 5000)
    public void writeRaspberryPiData(){
        logger.info("Insert raspberrypi stats");
        if(isUnix){
            try {
                raspberryPiDataService.save(new RaspberryPiData(
                        systemInformationService.getCpuTemperature(),
                        systemInformationService.getCpuVoltage(),
                        systemInformationService.getMemoryUsed(),
                        systemInformationService.getMemoryFree(),
                        systemInformationService.getMemoryBuffers(),
                        systemInformationService.getMemoryVoltageSDRam_C(),
                        systemInformationService.getMemoryVoltageSDRam_I(),
                        systemInformationService.getMemoryVoltageSDRam_P()
                        ));
            } catch (Exception e) {
                logger.error("writeRaspberryPiData error",e);
            }
        }

    }

}

Base Repo:

@NoRepositoryBean
public interface BaseRepo<T, ID extends Serializable> extends JpaRepository<T, ID>, JpaSpecificationExecutor<T> {
}

Right now I don't find the error in the log, but in the past it crashed with SIGILL error. I will edit this post after finding the log.

I found the log with error from JVM:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x741db028, pid=1918, tid=1620767840
#
# JRE version: Java(TM) SE Runtime Environment (8.0-b132) (build 1.8.0-b132)
# Java VM: Java HotSpot(TM) Server VM (25.0-b70 mixed mode linux-arm )
# Problematic frame:
# J 1635 C2 com.mysql.jdbc.ResultSetImpl.<init>(JJLcom/mysql/jdbc/MySQLConnection;Lcom/mysql/jdbc/StatementImpl;)V (314 bytes) @ 0x741db028 [0x741dae20+0x208]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x63003c00):  JavaThread "pool-2-thread-2" [_thread_in_Java, id=1967, stack(0x60960000,0x609b0000)]

siginfo:si_signo=SIGILL: si_errno=0, si_code=1 (ILL_ILLOPC), si_addr=0x741db028

Registers:
  r0  = 0x71897568
  r1  = 0x00000008
  r2  = 0x00000000
  r3  = 0x00000000
  r4  = 0x62922d40
  r5  = 0x00000368
  r6  = 0x693d65e8
  r7  = 0x62922d40
  r8  = 0x73a51000
  r9  = 0x66c13e30
  r10 = 0x63003c00
  fp  = 0x693d65e8
  r12 = 0x718974b8
  sp  = 0x609adfa0
  lr  = 0x73e74bb0
  pc  = 0x741db028
  cpsr = 0x60000010

Top of Stack: (sp=0x609adfa0)
0x609adfa0:   6e614d79 3d74696d 00000007 00000000
0x609adfb0:   609ae008 73e74bb0 00000000 00000000
0x609adfc0:   693d65e8 00000000 00000000 00000000
0x609adfd0:   693d65e8 00000000 00000000 00000000
0x609adfe0:   00000000 718974b8 609adfe8 629d07b7
0x609adff0:   609ae028 629d4420 602ed004 629d0800
0x609ae000:   609adfcc 609ae010 609ae04c 73e74bb0
0x609ae010:   00000000 693d65e8 00000000 00000000 

Instructions: (pc=0x741db028)
0x741db008:   e5d45008 e59cb05c e5cc5044 f57ff05f
0x741db018:   e59b4004 e1540007 1a00003a e3005368
0x741db028:   e18b40d5 e59c605c e1cc40f8 f57ff05f
0x741db038:   e5964004 e1540007 1a000036 e5964478 

Register to memory mapping:

  r0  = 0x71897568
0x71897568 is an oop
[Lcom.mysql.jdbc.Field; 
 - klass: 'com/mysql/jdbc/Field'[]
 - length: 0

  r1  = 0x00000008
0x00000008 is an unknown value

  r2  = 0x00000000
0x00000000 is an unknown value

  r3  = 0x00000000
0x00000000 is an unknown value

  r4  = 0x62922d40
0x62922d40 is an unknown value

  r5  = 0x00000368
0x00000368 is an unknown value

  r6  = 0x693d65e8
0x693d65e8 is an oop
com.mysql.jdbc.JDBC4Connection 
 - klass: 'com/mysql/jdbc/JDBC4Connection'

  r7  = 0x62922d40
0x62922d40 is an unknown value

  r8  = 0x73a51000
0x73a51000 is an unallocated location in the heap

  r9  = 0x66c13e30
0x66c13e30 is an unknown value

  r10 = 0x63003c00
0x63003c00 is a thread

  fp  = 0x693d65e8
0x693d65e8 is an oop
com.mysql.jdbc.JDBC4Connection 
 - klass: 'com/mysql/jdbc/JDBC4Connection'

  r12 = 0x718974b8
0x718974b8 is an oop
com.mysql.jdbc.JDBC42ResultSet 
 - klass: 'com/mysql/jdbc/JDBC42ResultSet'

  sp  = 0x609adfa0
0x609adfa0 is pointing into the stack for thread: 0x63003c00

  lr  = 0x73e74bb0
0x73e74bb0 is at code_begin+2176 in an Interpreter codelet
invoke return entry points  [0x73e74330, 0x73e74cc0]  2448 bytes

  pc  = 0x741db028
0x741db028 is at entry_point+520 in (nmethod*)0x741dad08



Stack: [0x60960000,0x609b0000],  sp=0x609adfa0,  free space=311k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 1635 C2 com.mysql.jdbc.ResultSetImpl.<init>(JJLcom/mysql/jdbc/MySQLConnection;Lcom/mysql/jdbc/StatementImpl;)V (314 bytes) @ 0x741db028 [0x741dae20+0x208]
j  com.mysql.jdbc.JDBC4ResultSet.<init>(JJLcom/mysql/jdbc/MySQLConnection;Lcom/mysql/jdbc/StatementImpl;)V+7
j  com.mysql.jdbc.JDBC42ResultSet.<init>(JJLcom/mysql/jdbc/MySQLConnection;Lcom/mysql/jdbc/StatementImpl;)V+7
j  sun.reflect.GeneratedConstructorAccessor17.newInstance([Ljava/lang/Object;)Ljava/lang/Object;+242
J 1634 C2 com.mysql.jdbc.ResultSetImpl.getInstance(JJLcom/mysql/jdbc/MySQLConnection;Lcom/mysql/jdbc/StatementImpl;)Lcom/mysql/jdbc/ResultSetImpl; (65 bytes) @ 0x741e0260 [0x741dff70+0x2f0]
j  com.mysql.jdbc.MysqlIO.buildResultSetWithUpdates(Lcom/mysql/jdbc/StatementImpl;Lcom/mysql/jdbc/Buffer;)Lcom/mysql/jdbc/ResultSetImpl;+170
j  com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(Lcom/mysql/jdbc/StatementImpl;IIIZLjava/lang/String;Lcom/mysql/jdbc/Buffer;ZJ[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetImpl;+18
j  com.mysql.jdbc.MysqlIO.readAllResults(Lcom/mysql/jdbc/StatementImpl;IIIZLjava/lang/String;Lcom/mysql/jdbc/Buffer;ZJ[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetImpl;+30
j  com.mysql.jdbc.MysqlIO.sqlQueryDirect(Lcom/mysql/jdbc/StatementImpl;Ljava/lang/String;Ljava/lang/String;Lcom/mysql/jdbc/Buffer;IIIZLjava/lang/String;[Lcom/mysql/jdbc/Field;)Lcom/mysql/jdbc/ResultSetInternalMethods;+806
J 1596 C2 com.mysql.jdbc.ConnectionImpl.execSQL(Lcom/mysql/jdbc/StatementImpl;Ljava/lang/String;ILcom/mysql/jdbc/Buffer;IIZLjava/lang/String;[Lcom/mysql/jdbc/Field;Z)Lcom/mysql/jdbc/ResultSetInternalMethods; (426 bytes) @ 0x741cb6d8 [0x741cb3f0+0x2e8]
j  com.mysql.jdbc.ConnectionImpl.setAutoCommit(Z)V+147
j  org.apache.commons.dbcp2.DelegatingConnection.setAutoCommit(Z)V+9
j  org.apache.commons.dbcp2.DelegatingConnection.setAutoCommit(Z)V+9
j  org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin()V+13
j  org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin()V+12
j  org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin()V+8
j  org.hibernate.engine.transaction.internal.TransactionImpl.begin()V+47
j  org.hibernate.internal.SessionImpl.beginTransaction()Lorg/hibernate/Transaction;+41
j  org.hibernate.jpa.internal.TransactionImpl.begin()V+42
j  org.springframework.orm.jpa.DefaultJpaDialect.beginTransaction(Ljavax/persistence/EntityManager;Lorg/springframework/transaction/TransactionDefinition;)Ljava/lang/Object;+26
j  org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(Ljavax/persistence/EntityManager;Lorg/springframework/transaction/TransactionDefinition;)Ljava/lang/Object;+34
j  org.springframework.orm.jpa.JpaTransactionManager.doBegin(Ljava/lang/Object;Lorg/springframework/transaction/TransactionDefinition;)V+148
j  org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(Lorg/springframework/transaction/TransactionDefinition;)Lorg/springframework/transaction/TransactionStatus;+204
j  org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(Lorg/springframework/transaction/PlatformTransactionManager;Lorg/springframework/transaction/interceptor/TransactionAttribute;Ljava/lang/String;)Lorg/springframework/transaction/interceptor/TransactionAspectSupport$TransactionInfo;+37
j  org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(Ljava/lang/reflect/Method;Ljava/lang/Class;Lorg/springframework/transaction/interceptor/TransactionAspectSupport$InvocationCallback;)Ljava/lang/Object;+49
j  org.springframework.transaction.interceptor.TransactionInterceptor.invoke(Lorg/aopalliance/intercept/MethodInvocation;)Ljava/lang/Object;+40
J 1449 C2 org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()Ljava/lang/Object; (107 bytes) @ 0x74187c50 [0x74187b90+0xc0]
j  org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(Lorg/aopalliance/intercept/MethodInvocation;)Ljava/lang/Object;+1
J 1449 C2 org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()Ljava/lang/Object; (107 bytes) @ 0x74187c50 [0x74187b90+0xc0]
j  org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(Lorg/aopalliance/intercept/MethodInvocation;)Ljava/lang/Object;+89
J 1449 C2 org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()Ljava/lang/Object; (107 bytes) @ 0x74187c50 [0x74187b90+0xc0]
j  org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Lorg/aopalliance/intercept/MethodInvocation;)Ljava/lang/Object;+18
J 1449 C2 org.springframework.aop.framework.ReflectiveMethodInvocation.proceed()Ljava/lang/Object; (107 bytes) @ 0x74187c50 [0x74187b90+0xc0]
j  org.springframework.aop.framework.JdkDynamicAopProxy.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;+321
j  com.sun.proxy.$Proxy67.save(Ljava/lang/Object;)Ljava/lang/Object;+16
j  ro.i2cWebAppController.app.serv.BaseServiceImpl.save(Lro/i2cWebAppController/app/model/BaseModel;)Lro/i2cWebAppController/app/model/BaseModel;+5
j  ro.i2cWebAppController.app.schedule.RaspberryPiScheduler.writeRaspberryPiData()V+116
j  sun.reflect.GeneratedMethodAccessor64.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+36
J 580 C2 java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (62 bytes) @ 0x73ffafac [0x73ffaf10+0x9c]
j  org.springframework.scheduling.support.ScheduledMethodRunnable.run()V+19
j  org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run()V+4
j  java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;+4
j  java.util.concurrent.FutureTask.runAndReset()Z+47
j  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Ljava/util/concurrent/ScheduledThreadPoolExecutor$ScheduledFutureTask;)Z+1
j  java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V+37
j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
C  0x609aed68

And i found a very strange behaviour. After following the advices from comments, i added the following lines in tomcat starting script :

Environment='CATALINA_OPTS=-Xms64M -Xmx192M -server -XX:+UseParallelGC -XX:+UnlockCommercialFeatures -XX:+FlightRecorder'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Dcom.sun.management.jmxremote.port=7091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false'

and i connected Java Mission Control and Flight Recording to raspberry's JVM. The strange thing is that the JVM didn't crash after 12 hours of monitoring, everything was working ok. After restarting the server with "XX:+UnlockCommercialFeatures -XX:+FlightRecorder" and "-Dcom.sun.management.jmxremote.port=7091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" removerd, JVM crashed after 7 hours.

And with Mission Control and Flight Recording connected to JVM i also noticed that Committed Heap is allways rising but never dropping.

Java memory Heap Diagram

Answer 1

Problem solved after Java update from 1.8.0 to 1.8.0_65.

Thank you for tips.