将2D矩阵存储在CSV中：java.lang.OutOfMemoryError

Question

I wrote the following function for calculating cost matrix and storing it in CSV file:

private static void calculateCostMatrix()
{
    int len = _POIs.size();
    CostMatrix = new double[len][len];
    for (int i=0; i<len; i++)
    {
        int ic = _POIs.get(i).getId();
        for (int j=i; j<len; j++)
        {
            int jc = _POIs.get(j).getId();
            double dist = euclideandist(_POIs.get(i).getLat(),_POIs.get(i).getLon(),
                                    _POIs.get(j).getLat(),_POIs.get(j).getLon());
            CostMatrix[ic][jc] = dist;
            CostMatrix[jc][ic] = dist;
        }           
    }

    // Save in CSV
    try
    {
        String NEW_LINE = System.getProperty("line.separator");
        File file = new File("CostMatrix.csv");
        FileWriter fw = new FileWriter(file.getAbsoluteFile()); 
        BufferedWriter bw = new BufferedWriter(fw);
        StringBuilder sb = new StringBuilder();;
        for (double[] row : CostMatrix) 
        {
            for (double d : row)
            {
                 sb.append(d);
                 sb.append(",");
            }
            sb.append(NEW_LINE);
        }
        bw.write(sb.toString());
        bw.close();
        }

        bw.close();
    } catch (IOException e) {
        e.printStackTrace();
    }
}

However, obviously, due to large number of elements (200,000 x 200,000) I get the message java.lang.OutOfMemoryError: Java heap space .

So, instead of pre-allocating a space for CostMAtrix, I now want to directly calculate and store each cell in CSV file. How can I do this in the proper way?

Answer 1

I would do it differently. Since the problem is memory, you could use a temporary binary file on disk instead of an array to store the result. Create this temporary files of the right size, then seek through it to store the double results (in binary form).

Once you are done processing the whole matrix in binary, parse the file to convert it to CSV.

Otherwise, you'll have to sort your POIs by value and keep their index as a reference for the CSV row/column loop.