Python根据匹配的键/值对减少字典列表

Question

I have a list of dicts which specify flows (source to hop to destionation with their respective volume). Now i want to split these flows into link (eg (source to hop with volume, hop to destination with volume) and merge all duplicate links together by summing up their volumes.

Since I'm new to python I'm wondering what a good approach would be. My first approach would be to loop through all flows and nest a loop through all links inside and check if the links already exists.

But if I have millions of flows, that might become quite ineffienct and slow I guess.

My starting data looks like this:

flows = [
    {
        'source': 1,
        'hop': 2,
        'destination': 3,
        'volume': 100,
    },{
        'source': 1,
        'hop': 2,
        'destination': 4,
        'volume': 50,
    },{
        'source': 2,
        'hop': 2,
        'destination': 4,
        'volume': 200,
    },
]

What my result should be:

links = [
    {
        'source': 1,
        'hop': 2,
        'volume': 150,
    },{
        'hop': 2,
        'destination': 3,
        'volume': 100,
    },{
        'hop': 2,
        'destination': 4,
        'volume': 250,
    },{
        'source': 2,
        'hop': 2,
        'volume': 200,
    },
]

Thanks a lot for your help!

Answer 1

You can collect the links to two different dictionaries, one between source & hop and another one between hop & destination. Then you can easily create the result list separately from both of the dicts. Below Counter is used which is dict like object with 0 as default value:

import pprint
from collections import Counter

flows = [
    {
        'source': 1,
        'hop': 2,
        'destination': 3,
        'volume': 100.5,
    },{
        'source': 1,
        'hop': 2,
        'destination': 4,
        'volume': 50,
    },{
        'source': 2,
        'hop': 2,
        'destination': 4,
        'volume': 200.7,
    },
]

sources = Counter()
hops = Counter()

for f in flows:
    sources[f['source'], f['hop']] += f['volume']
    hops[f['hop'], f['destination']] += f['volume']

res = [{'source': source, 'hop': hop, 'volume': vol} for (source, hop), vol in sources.items()]
res.extend([{'hop': hop, 'destination': dest, 'volume': vol} for (hop, dest), vol in hops.items()])
pprint.pprint(res)

Output:

[{'hop': 2, 'source': 1, 'volume': 150.5},
 {'hop': 2, 'source': 2, 'volume': 200.7},
 {'destination': 3, 'hop': 2, 'volume': 100.5},
 {'destination': 4, 'hop': 2, 'volume': 250.7}]

Above will run in O(n) time so it should work with millions of flows provided you have enough memory.

Answer 2

pseudo algorithm:

1. create an empty result list/set/dictionary
2. loop over de flows list
3. split up each single flow into 2 links
4. for each of these 2 links test if they are already in the result list (based on the 2 nodes).
5. if not: add them. if yes: upgrade the volume of the one already in the list.