Python: Memory Error with global efficiency (networkx)

Question

I am dealing with a 100x100 grid network. I want to determine its global efficiency to see how efficiently information is exchanged within it.

I am using a bespoke function for computing the efficiency, and then I apply it to my network.

However, I run into a Memory Error which points at the line where the function is called (last line). Does this depend on how much RAM Python is using? How can I fix this?

The code is as follows:

from __future__ import print_function, division
import numpy
from numpy import *
import networkx as nx
import matplotlib.pyplot as plt
import csv
from collections import *
import os
import glob
from collections import OrderedDict 

def global_efficiency(G, weight=None):
    N = len(G)
    if N < 2:
        return 0   
    inv_lengths = []
    for node in G:
        if weight is None:
            lengths = nx.single_source_shortest_path_length(G, node)
        else:
            lengths=nx.single_source_dijkstra_path_length(G,node,weight=weight)

        inv = [1/x for x in lengths.values() if x is not 0]
        inv_lengths.extend(inv)

    return sum(inv_lengths)/(N*(N-1))

N=100
G=nx.grid_2d_graph(N,N)
pos = dict( (n, n) for n in G.nodes() )
labels = dict( ((i, j), i + (N-1-j) * N ) for i, j in G.nodes() )
nx.relabel_nodes(G,labels,False)
inds=labels.keys()
vals=labels.values()
inds.sort()
vals.sort()
pos2=dict(zip(vals,inds))
nx.draw_networkx(G, pos=pos2, with_labels=False, node_size = 10)
eff=global_efficiency(G)

Answer 1

I think I know why you have a memory error. Keep all the lengths of all shortest paths for each node can yield to a very huge list inv_lengths .

I suggest the equivalent modification:

def global_efficiency(G, weight=None):
    N = len(G)
    if N < 2:
        return 0   
    inv_lengths = []
    for node in G:
        if weight is None:
            lengths = nx.single_source_shortest_path_length(G, node)
        else:
            lengths=nx.single_source_dijkstra_path_length(G,node,weight=weight)

        inv = [1/x for x in lengths.values() if x is not 0]

        # Changes here
        inv_sum = sum(inv)
        inv_lengths.append(inv_sum)  # add results, one per node

    return sum(inv_lengths)/(N*(N-1))

It gives the same result (I checked).