C＃中链接的2D矩阵

Question

I need to implement this scenario in C#:

The matrix will be very large, maybe 10000x10000 or larger. I will use this for distance matrix in hierarchical clustering algorithm. In every iteration of the algorithm the matrix should be updated (joining 2 rows into 1 and 2 columns into 1). If I use simple double[,] or double[][] matrix this operations will be very "expensive". Please, can anyone suggest C# implementation of this scenario?

Answer 1

Do you have a algorithm at the moment? And what do you mean by expensive? Memory or time expensive? If memory expensive: There is not much you can do in c#. But you can consider executing the calculation inside a database using temporary objects. If time expensive: You can use parallelism to join columns and rows.

But beside that I think a simple double[,] array is the fastest and memory sparing way you can get in c#, because accessing the array values is an o(1) operation and arrays have a least amount of memory and management overhead (compared to lists and dictionaries).

Answer 2

As mentioned above, a basic double[,] is going to be the most effective way of handling this in C#.

Remember that C# sits of top of managed memory, and as such you have less fine grain control over low level (in terms of memory) operations in contrast to something like basic C. Creating your own objects in C# to add functionality will only use more memory in this scenario, and likely slow the algorithm down as well.

If you have yet to pick an algorithm, CURE seems to be a good bet. The choice of algorithm may affect your data structure choice, but that's not likely.

You will find that the algorithm determines the theoretical limits of 'cost' at any rate. For example you will read that for CURE, you are bound by a O(n2 log n) running time, and O(n) memory use.

I hope this helps. If you can provide more detail, we might be able to assist further!

N.

Answer 3

It's not possible to 'merge' two rows or two columns, you'd have to copy the whole matrix into a new, smaller one, which is indeed unacceptably expensive.

You should probably just add the values in one row to the previous and then ignore the values, acting like they where removed.

the arrays of arrays: double[][] is actually faster than double[,]. But takes more memory.

The whole array merging thing might not be needed if you change the algoritm a bit, but this might help u:

    public static void MergeMatrix()
    {
        int size = 100;
        // Initialize the matrix
        double[,] matrix = new double[size, size];
        for (int i = 0; i < size; i++)
            for (int j = 0; j < size; j++)
                matrix[i, j] = ((double)i) + (j / 100.0);

        int rowMergeCount = 0, colMergeCount = 0;
        // Merge last row.
        for (int i = 0; i < size; i++)
            matrix[size - rowMergeCount - 2, i] += matrix[size - rowMergeCount - 1, i];
        rowMergeCount++;
        // Merge last column.
        for (int i = 0; i < size; i++)
            matrix[i, size - colMergeCount - 2] += matrix[i, size - colMergeCount - 1];
        colMergeCount++;

        // Read the newly merged values.
        int newWidth = size - rowMergeCount, newHeight = size - colMergeCount;
        double[,] smaller = new double[newWidth, newHeight];
        for (int i = 0; i < newWidth; i++)
            for (int j = 0; j < newHeight; j++)
                smaller[i, j] = matrix[i, j];

        List<int> rowsMerged = new List<int>(), colsMerged = new List<int>();
        // Merging row at random position.
        rowsMerged.Add(15);
        int target = rowsMerged[rowMergeCount - 1];
        int source = rowsMerged[rowMergeCount - 1] + 1;
        // Still using the original matrix since it's values are still usefull.
        for (int i = 0; i < size; i++)
            matrix[target, i] += matrix[source, i];
        rowMergeCount++;

        // Merging col at random position.
        colsMerged.Add(37);
        target = colsMerged[colMergeCount - 1];
        source = colsMerged[colMergeCount - 1] + 1;
        for (int i = 0; i < size; i++)
            matrix[i, target] += matrix[i, source];
        colMergeCount++;

        newWidth = size - rowMergeCount;
        newHeight = size - colMergeCount;
        smaller = new double[newWidth, newHeight];
        for (int i = 0, j = 0; i < newWidth && j < size; i++, j++)
        {
            for (int k = 0, m = 0; k < newHeight && m < size; k++, m++)
            {
                smaller[i, k] = matrix[j, m];
                Console.Write(matrix[j, m].ToString("00.00") + " ");

                // So merging columns is more expensive because we have to check for it more often while reading.
                if (colsMerged.Contains(m)) m++;
            }

            if (rowsMerged.Contains(j)) j++;
            Console.WriteLine();
        }

        Console.Read();
    }

Answer 4

In this code I use two 1D helper lists to calculate the index into a big array containing the data. Deleting rows/columns is really cheap since I only need to remove that index from the helper-lists. But of course the memory in the big array remains, ie depending on your usage you have a memory-leak.

public class Matrix
{
    double[] data;
    List<int> cols;
    List<int> rows;

    private int GetIndex(int x,int y)
    {
        return rows[y]+cols[x];
    }

    public double this[int x,int y]
    {
        get{return data[GetIndex(x,y)];}
        set{data[GetIndex(x,y)]=value;} 
    }

    public void DeleteColumn(int x)
    {
        cols.RemoveAt(x);
    }

    public void DeleteRow(int y)
    {
        rows.RemoveAt(y);
    }

    public Matrix(int width,int height)
    {
        cols=new List<int>(Enumerable.Range(0,width));
        rows=new List<int>(Enumerable.Range(0,height).Select(i=>i*width));
        data=new double[width*height];
    }
}

Answer 5

Hm, to me this looks like a simple binary tree. The left node represents the next value in a row and the right node represents the column.

So it should be easy to iterate rows and columns and combine them.

Answer 6

Thank you for the answers.

At the moment I'm using this solution:

public class NodeMatrix
{

    public NodeMatrix Right { get; set;}
    public NodeMatrix Left { get; set; }
    public NodeMatrix Up { get; set; }
    public NodeMatrix Down { get; set; }
    public int I  { get; set; }
    public int J  { get; set; }
    public double Data { get; set; }

    public NodeMatrix(int I, int J, double Data)
    {
        this.I = I;
        this.J = J;
        this.Data = Data;
    }
}

List<NodeMatrix> list = new List<NodeMatrix>(10000);

Then I'm building the connections between the nodes. After that the matrix is ready.

This will use more memory, but operations like adding rows and columns, joining rows and columns I think will be far more faster.