在python中创建一个邻接矩阵

Question

I want to load CSV or text file of signed (weighted) graph and create an adjacency matrix. The CSV file contains three columns named "FromNodeId", "ToNodeId" and "Sign". The code I used is as follows:

G = nx.read_edgelist('soc-sign-epinions.txt', data = [('Sign', int)])
#print(G.edges(data = True))

A = nx.adjacency_matrix(G)
print(A.todense())

I encountered the following error

ValueError: array is too big; `arr.size * arr.dtype.itemsize` is larger than 
the maximum possible size

How can I solve this problem? Please suggest me a way to create the adjacency matrix.

Answer 1

The memory needed to store a big matrix can easily get out of hand, which is why nx.adjacency_matrix(G) returns a "sparse matrix" which is stored more efficiently (exploiting that many entries will be 0).

Since your graph has 131000 vertices, the whole adjacency matrix will use around 131000^2 * 24 bytes (an integer takes 24 bytes of memory in python), which is about 400GB. However, your graph has less than 0.01% of all edges, in other words it is very sparse and sparse matrices will work for you.

In order to get the sparse matrix, just use A = nx.adjacency_matrix(G) without calling A.todense() after it (this tries to store it normally again).

There is an inbuild function of scipy.sparse to efficiently save and load sparse matrices, see here . For example, to save your sparse matrix A, use

scipy.sparse.save_npz('filename.npz', A)

If it is important for you to use txt or CSV, you will have to do it manually. This can be done by iterating through every row of your matrix and writing these one by one to your file:

for i in range(A.shape[0]): row = A.getrow(i).todense() [write row to file using your preferred method]

This might take a few minutes to run, but should work (I tested with a path of the same size).