public void filterLogins() throws IOException, SQLException{
for(int i = 0; i<200; ++i){
BufferedReader bufferedReader = new BufferedReader(new FileReader(folder + String.format("\\data\\part-%05d", i)));
long prev_id = 0;
boolean contains = false;
while(bufferedReader.ready()){ //very big file
String line = bufferedReader.readLine();
Login login = new Login(line);
if ( login.userId == prev_id && !contains )
continue;
if ( samples.contains(login.userId) ){
mysql.execute("INSERT INTO ..."); // i think it doesn't matter in this case
contains = true;
}else{
contains = false;
}
prev_id = login.userId;
}
bufferedReader.close();
System.out.println((double)i/2.0);
}
}
This function works long time, because data is more big files. 2 hours ago this is crushess with OutOfMemory Exception
mysql is instance of
public class MySQLHandler {
private Connection connection = null;
MySQLHandler() throws ClassNotFoundException, SQLException{
try{
Class.forName("com.mysql.jdbc.Driver");
connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/database", "root", "");
}catch (ClassNotFoundException e){
System.out.println("Failed jdbc driver load.");
throw e;
}
}
public void close() throws SQLException{
if ( connection != null)
connection.close();
}
public boolean execute(String sql) throws SQLException{
Statement statement = connection.createStatement();
return statement.execute(sql);
}
}
Login it is just class with data. (id, name, value, something else).
Seems, you are creating Statement each time and not closing the statement in your loop which will leak memory . close the statement once execute is completed.
public boolean execute(String sql) throws SQLException{
Statement statement = connection.createStatement();
return statement.execute(sql);
}
Like
public boolean execute(String sql) throws SQLException{
Statement statement = null;
try {
statement = connection.createStatement();
return statement.execute(sql);
}finaly{
if (statement != null) statement.close();
}
UPDATE
As @Holger mentioned in comment, if you are using JDK 7 and higher then you can use try-with-resources like below
public boolean execute(String sql) throws SQLException{
try(Statement s=connection.createStatement()) {
return s.execute(sql);
}
}
Try increasing the max heap size of your JVM eg:
-Xms<size> set initial Java heap size
-Xmx<size> set maximum Java heap size
-Xss<size> set java thread stack size
java -Xms16m -Xmx64m ClassName
Also optimzie your code and reuse variables and close/cleanup after use. For eg: Move these outside the loop: BufferedReader bufferedReader, String line and Login login
public void filterLogins() throws IOException, SQLException{
BufferedReader bufferedReader;
long prev_id;
boolean contains;
String line;
Login login;
for(int i = 0; i<200; ++i){
bufferedReader = new BufferedReader(new FileReader(folder + String.format("\\data\\part-%05d", i)));
prev_id = 0;
contains = false;
while(bufferedReader.ready()){ //very big file
line = bufferedReader.readLine();
login = new Login(line);
if ( login.userId == prev_id && !contains )
continue;
if ( samples.contains(login.userId) ){
mysql.execute("INSERT INTO ..."); // i think it doesn't matter in this case
contains = true;
}else{
contains = false;
}
prev_id = login.userId;
}
bufferedReader.close();
System.out.println((double)i/2.0);
}
}
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.