简体   繁体   中英

Why does this function cause a memory leak?

public void filterLogins() throws IOException, SQLException{
    for(int i = 0; i<200; ++i){
        BufferedReader bufferedReader = new BufferedReader(new FileReader(folder + String.format("\\data\\part-%05d", i)));
        long prev_id = 0;
        boolean contains = false;
        while(bufferedReader.ready()){ //very big file
            String line = bufferedReader.readLine();
            Login login = new Login(line);
            if ( login.userId == prev_id && !contains )
                continue;
            if ( samples.contains(login.userId) ){
                mysql.execute("INSERT INTO ..."); // i think it doesn't matter in this case
                contains = true;
            }else{
                contains = false;
            }
            prev_id = login.userId;
        }
        bufferedReader.close();
        System.out.println((double)i/2.0);
    }
}

This function works long time, because data is more big files. 2 hours ago this is crushess with OutOfMemory Exception

mysql is instance of

public class MySQLHandler {
private Connection connection = null;
MySQLHandler() throws ClassNotFoundException, SQLException{
    try{
        Class.forName("com.mysql.jdbc.Driver");
        connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/database", "root", "");
    }catch (ClassNotFoundException e){
        System.out.println("Failed jdbc driver load.");
        throw e;
    }
}


public void close() throws SQLException{
       if ( connection != null)
           connection.close();
   }
   public boolean execute(String sql) throws SQLException{
       Statement statement = connection.createStatement();
       return statement.execute(sql);
   }
}

Login it is just class with data. (id, name, value, something else).

Seems, you are creating Statement each time and not closing the statement in your loop which will leak memory . close the statement once execute is completed.

 public boolean execute(String sql) throws SQLException{
       Statement statement = connection.createStatement();
       return statement.execute(sql);
   }

Like

 public boolean execute(String sql) throws SQLException{
     Statement statement = null;
    try {
        statement = connection.createStatement();
        return statement.execute(sql);
    }finaly{
         if (statement != null) statement.close();
     }

UPDATE

As @Holger mentioned in comment, if you are using JDK 7 and higher then you can use try-with-resources like below

public boolean execute(String sql) throws SQLException{
       try(Statement s=connection.createStatement()) {
          return s.execute(sql);
       }
 }

Try increasing the max heap size of your JVM eg:

-Xms<size>        set initial Java heap size
-Xmx<size>        set maximum Java heap size
-Xss<size>        set java thread stack size

java -Xms16m -Xmx64m ClassName

Also optimzie your code and reuse variables and close/cleanup after use. For eg: Move these outside the loop: BufferedReader bufferedReader, String line and Login login

public void filterLogins() throws IOException, SQLException{
    BufferedReader bufferedReader;
    long prev_id;
    boolean contains;
    String line;
    Login login;
    for(int i = 0; i<200; ++i){
        bufferedReader = new BufferedReader(new FileReader(folder + String.format("\\data\\part-%05d", i)));
        prev_id = 0;
        contains = false;
        while(bufferedReader.ready()){ //very big file
            line = bufferedReader.readLine();
            login = new Login(line);
            if ( login.userId == prev_id && !contains )
                continue;
            if ( samples.contains(login.userId) ){
                mysql.execute("INSERT INTO ..."); // i think it doesn't matter in this case
                contains = true;
            }else{
                contains = false;
            }
            prev_id = login.userId;
        }
        bufferedReader.close();
        System.out.println((double)i/2.0);
    }
}

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM