CSV Java文件读取和保存（在不同的ArrayList中）

Question

Ok mates, here is my code. I've got a problem, because "records.csv" is a file which cointains moreless 20 millions line, each one made of 4 fields separated with a ','.

As you can understand from the code, i'd like to have 4 Arraylists, each of them with all the values of a different field. The method after a while stop working (i think because to 'add' an element to the list, java has a pointer that have to tread all the arraylist before).

I need to solve, but i don't know how.

Suggestions?

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.util.ArrayList;

    public class RecordReader {
    static ArrayList<String> id = new ArrayList <String> ();
    static ArrayList<String> field1 = new ArrayList <String> ();
    static ArrayList<String> field2 = new ArrayList <String> ();
    static ArrayList<String> field3 = new ArrayList <String> ();



    public static void Reader () {
        try {
        FileReader filein = new FileReader("Y:/datasets/records.csv");
        String token="";
        String flag = "id";
        int index=0, next;

        do {
            next = filein.read();

            if (next != -1) {

                if (next !=',' && next !='\n') 
                    token = token + next;

                else if (next == ','){
                    if (flag.compareTo("id")==0) {id.add (index, token); flag = "field1";}
                    else if (flag.compareTo("field1")==0) {field1.add (index, token); token=""; flag = "field2";}
                    else if (flag.compareTo("field2")==0) {field2.add (index, token); token=""; flag = "field3";}
                }

                else if (next == '\n') { 
                    if (flag.compareTo("field3")==0) {field3.add (index, token); token=""; flag = "id"; index++;} 
                }

                char nextc = (char) next; 
                System.out.print(nextc); 
                }
        } while (next!=-1);

        filein.close();
        }
        catch (IOException e) { System.out.println ("ERRORE, birichino!"); }
    }
}

I have to do it all in once, the file is 711000 bytes.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.nio.CharBuffer.wrap(Unknown Source) at sun.nio.cs.StreamEncoder.implWrite(Unknown Source) at sun.nio.cs.StreamEncoder.write(Unknown Source) at java.io.OutputStreamWriter.write(Unknown Source) at java.io.BufferedWriter.flushBuffer(Unknown Source) at java.io.PrintStream.write(Unknown Source) at java.io.PrintStream.print(Unknown Source) at RecordReader.Reader(RecordReader.java:42) at prova.main(prova.java:26)

Answer 1

I have a couple of suggestions for you.

First, you don't need to have 4 separate ArrayLists , just one will do fine. Instead of using filein.read() , I would wrap your FileReader with a BufferedReader and use it to read the file line by line and add each line to a single ArrayList .

BufferedReader br = new BufferedReader(filein);
ArrayList<String> content = new ArrayList<String>();
String line = br.readLine();
while(line != null){
    //add lines to ArrayList
    content.add(line);
    line = br.readLine();
}

This will read the contents of the entire file into memory without the additional overhead of 3 extra ArrayLists .

Second, since your fields are separated by a , and (I'm assuming) always have the same number of fields, you can use the split() method to separate each line into an array of strings.

String[] record = content.get(index).split(",");
//record[0] = id
//record[1] = field1
//record[2] = field2
//record[3] = field3

Put the above into a loop and you can iterate over all of the file's contents. Since you know how the information is ordered, retrieving the information you want is trivial to do.

However, I will warn you that with a sufficiently large enough file (with multiple GB of data), eventually this approach will also fail.

Answer 2

Can you try running the application with -Xmx option as shown below

java -Xmx6g [javaclassfile]

I was able to resolve similar problem with this.