查找列表中的重复元素并平均对应元素

Question

I have a list of the following type in my code:

nodes = [[n1, [x1,y1,z1], [a1,b1,...]], [n2, [x2,y2,z2], [a2,b2,..]], ...]

The elements n identify a node with a certain coordinate [x,y,z], and these can be repeated trough the list. I wanted to average the [an,bn,...] lists for each node value n , such that I end up with a list nodes_averaged like [[[x1,y1,z1],[a1,b1,...]], [[xn,yn,zn],[an,bn,...]]]

For example, if I have the following list:

nodes= [[10, [6.5, 55.2, -10.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]], 
        [10, [6.5, 55.2, -10.0], [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]],
        [3, [4.3, 55.4, -15.0], [-0.0016, -0.00058, -0.0256, -7.07e-06, 0.00051, 0.0088]],
        [1, [8.7, 54.9, -15.0], [-0.0016, -0.00058, -0.0256, 1.2e-05, -0.00044, 0.0088]],
        [10, [6.5, 55.2, -10.0], [-0.0011, -0.00041, -0.027, -1.12e-05, -0.00043, 0.0038]],
        [3, [4.3, 55.4, -15.0], [-0.00113, -0.000413, -0.027, 2.84e-06, 0.00039, 0.00389]]]

I wanted to obtain (the order is not important):

nodes_avg= [[[6.5, 55.2, -10.0], [-0.00036666666666666667, -0.00013666666666666666, -0.009, -3.7333333333333333e-06, -0.00014333333333333334, 0.0012666666666666666]],
            [[4.3, 55.4, -15.0], [-0.001365, -0.0004965, -0.0263, -2.115e-06, 0.00045, 0.006345]], 
            [[8.7, 54.9, -15.0], [-0.0016, -0.00058, -0.0256, 1.2e-05, -0.00044, 0.0088]]]

So far my approach was to extract a list of the node's range:

noderng = [n1, n2,..., nn]

and using node_range=list(dict.fromkeys(node_range)) to eliminate repeated nodes, then iterating over both lists to match the elements n from the nodes list to the elements in the node_range list, like this:

noderng=[]
for l in nodes:
    noderng.append(l[0])
noderng=list(dict.fromkeys(noderng))

nodeavg={}
elposition={}

for n in noderng:
    nodeavg[n]=[[],[],[],[],[],[]]
    for l in nodes:
        if int(l[0])==int(n):
            for i in range(6):
                nodeavg[n][i].append(l[2][i])
            elposition[n]=l[1]

avg={}    
for key, strains in nodeavg.items():
    avg[key]=[]
    for ess in strains:
        avg[key].append(sum(ess)/len(ess))

nodes_avg=[]
for key ,value in avg.items():
    nodes_avg.append([elposition[key],value])

I get the desired result, the problem is that the nodes list can have hundreds of thousands of elements, and it'll take hours to do this operation. I have switched to using numpy arrays, and it does help slightly, but it only shaves off a couple of minutes, which is not a great help.

Is there a more efficient way to do this operation?

Answer 1

This is not an answer just a few suggestions:

1. You can use timer and cProfile to understand which part of your code takes more time and focus on that part.

2. another suggestion that has worked for me in the past is to create a dictionary from a set that includes your keys. In this way, you can reduce the hash time. This works if you know all the n values beforehand.

3. Check this to have a better understanding of the time complexity of different structs in Python.