读取双打矩阵的有效方法

Question

What is a very fast way to read in a clean matrix of all doubles (no missing elements on NAs are in this matrix). Most entries are non-zero doubles, maybe 30% are zero. The dimensions are around 1 million rows and 100 columns.

The function I am using is below. However it is quite slow for matrices over 1 gigabyte.

How can I do this faster? Would any of the following help: - Instead of saving as csv and reading that, try saving as a binary format or another format. - Transpose the matrix in the data file, then read column by column instead of reading row by row as the below function is doing. - Somehow serializing the matrix as a Java object for re-reads.

 private static Vector<Vector<Double>> readTXTFile(String csvFileName, int skipRows) throws IOException {
     String line = null;
     BufferedReader stream = null;
     Vector<Vector<Double>> csvData = new Vector<Vector<Double>>();

     try {
         stream = new BufferedReader(new FileReader(csvFileName));
         int count = 0;
         while ((line = stream.readLine()) != null) {
            count += 1;
            if(count <= skipRows) {
                continue;
            }
             String[] splitted = line.split(",");
             Vector<Double> dataLine = new Vector<Double>(splitted.length);
             for (String data : splitted) {
                 dataLine.add(Double.valueOf(data));
             }

            csvData.add(dataLine);
         }
     } finally {
         if (stream != null)
             stream.close();
     }

     return csvData;
 }

Answer 1

I changed your code to get rid of all of the creation of Vector and Double objects in favor of using a fix-sized matrix (which does assume you know or can calculate the number of rows and columns in the file ahead of time).

I threw 500,000 line files at it and was seeing about 25% improvement.

private static double[][] readTXTFile(String csvFileName, int skipRows) throws IOException {
    BufferedReader stream = null;
    int totalRows = 500000, totalColumns = 6;
    double[][] matrix = new double[totalRows][totalColumns];

    try {
        stream = new BufferedReader(new FileReader(csvFileName));
        for (int currentRow = 0; currentRow < totalRows; currentRow++) {
            String line = stream.readLine();
            if (currentRow <= skipRows) {
                continue;
            }
            String[] splitted = line.split(",");
            for (int currentColumn = 0; currentColumn < totalColumns; currentColumn++) {
                matrix[currentRow][currentColumn] = Double.parseDouble(splitted[currentColumn]);
            }
        }
    } finally {
        if (stream != null) {
            stream.close();
        }
    }
    return matrix;
}