read from csv file in java

Question

I have a huge CSV file.I have to read it and store in a database using java.below code only read about 2000 rows from that file and store it into database.why? note the below calculation is to change the seconds to minutes with approximation.dont think a lot about that

import com.opencsv.CSVReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.List;


public class ReadCSV {

    public static void main(String[] args) throws SQLException{

        MainController mc = new MainController();
        boolean status=false; 

        String csvFile = "C:/Users/Thanushiya/Desktop/mobios/internship/csvfile/csvfile/Master.csv";
        CSVReader  reader = null;
        List myList = new ArrayList();
        String[] row = null;
        int duration_m = 0;
        try {
            reader = new CSVReader (new FileReader(csvFile), ',', '\'', 17);
            myList = reader.readAll();
            int i=0;
                for (Object object : myList) {
                    row = (String[]) object;
                    float duration_float = Float.parseFloat(row[12]) / 60 ;
                    float duration_mod = duration_float % 1 ;

                    if(Integer.parseInt(row[12]) <= 60){
                        duration_m = 1;
                    }
                    else{
                        if(duration_mod == 0 ){
                            duration_m = (int) duration_float;
                        }
                        else{
                            duration_m = ((int) duration_float) + 1;
                        }
                    }
                    String query = "INSERT INTO master(msisdn,serv,start,end,ringwithdurartion,duration_s,status,cid,duration_m) values ('"+row[0]+"','"+row[4]+"','"+row[8]+"','"+row[10]+"','"+row[11]+"','"+row[12]+"','"+row[13]+"','"+row[15]+"','"+duration_m+"')";
                    status=mc.insertData(query);

                }//end for

        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            if (reader != null) {
                try {
                    reader.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
        }

        System.out.println("Done");
    }

}

Answer 1

With readAll() you're loading (or at least trying to load) the entire file into memory at once. That will severely limit the number of lines you can read before running out of memory. As indicated on the OpenCSV homepage , there's also an iterator that reads line per line:

 CSVReader reader = new CSVReader(new FileReader("C:/Users/Thanushiya/Desktop/mobios/internship/csvfile/csvfile/Master.csv"), ',', '\'', 17);
 String [] nextLine;
 while ((nextLine = reader.readNext()) != null) {
    // put the code from your for loop here
 }