Remove duplicates of IP addresses by netmask

Question

I have got an Array:

[192.0.0.3, 0.0.0.0 , 192.0.10.24, ...]

With IP addresses and i want to remove duplicates for the /16 netmasks, so i got 192.0.0.3 but 192.0.10.24 will be removed (i don't mind which one of them, it would also be okay if the first one is removed).

My first thoughts were to use a regex to cast the netmask and remove every IP address which matches the then generated patttern.

Is there an easier way?

Answer 1

You could remove duplicates using a set, with the keys being tuples of the first two parts:

>>> ips = ["192.0.0.3", "0.0.0.0", "192.0.10.24"]
>>> seen = set()
>>> for ip in ips:
...     key = tuple(ip.split(".")[:2])
...     if key not in seen:
...         print(ip)
...         seen.add(key)
... 
192.0.0.3
0.0.0.0

Or alternatively using the ipaddress module:

>>> from ipaddress import ip_network
>>> ips = ["192.0.0.3", "0.0.0.0", "192.0.10.24"]
>>> seen = set()
>>> for ip in ips:
...     key = ip_network(ip + "/16", strict=False)
...     if key not in seen:
...         print(ip)
...         seen.add(key)
... 
192.0.0.3
0.0.0.0

Answer 2

You could use a dictionary:

>>> res = {}
>>> for ip in ["192.0.0.3", "0.0.0.0", "192.0.10.24"]:
...    res[tuple(ip.split('.',2)[0:2])]=ip
>>> res.values()
['0.0.0.0', '192.0.10.24']

If you need the first occurence rather than the last one, a quick and dirty solution is to reverse the list first:

>>> res = {}
>>> for ip in reversed(["192.0.0.3", "0.0.0.0", "192.0.10.24"]):
...    res[tuple(ip.split('.',2)[0:2])]=ip
>>> res.values()
['0.0.0.0', '192.0.0.3']

Example with ipaddress as @eugne s suggests:

>>> import ipaddress
>>> res = {}
>>> for ip in [u"192.0.0.3", u"0.0.0.0", u"192.0.10.24"]:
...    res[ipaddress.ip_network(ip + "/16", strict=False)]=ip
>>> res.values()
[u'192.0.10.24', u'0.0.0.0']