有没有更好的方法来迭代两个列表来查找python中的项之间的关系？

Question

I mock a ip list and a subnet dict as input:

# ip address list
ip_list = [
'192.168.1.151', '192.168.10.191', '192.168.6.127', 
'192.168.2.227', '192.168.2.5', '192.168.3.237', 
'192.168.6.188', '192.168.7.209', '192.168.9.10',
# Edited: add some /28, /16 case
'192.168.12.39', '192.168.12.58', '10.63.11.1', '10.63.102.69',
]

# subnet dict
netsets = {
'192.168.1.0/24': 'subnet-A',     # {subnet: subnet's name} 
'192.168.10.0/24': 'subnet-B', 
'192.168.2.0/24': 'subnet-C', 
'192.168.3.0/24': 'subnet-C',
'192.168.6.0/24': 'subnet-D', 
'192.168.7.0/24': 'subnet-D', 
'192.168.9.0/24': 'subnet-E',
# Edited: add some /28, /16 case
'192.168.12.32/28': 'subnet-F',
'192.168.12.48/28': 'subnet-G',
'10.63.0.0/16': 'subnet-I',
}

and then each ip address in ip_list need to find the name of subnet.

We assume that each ip address can find the corresponding subnet in netsets .

Ouput like this:

192.168.1.151   subnet-A
192.168.10.191  subnet-B
192.168.6.127   subnet-D
192.168.2.227   subnet-C
192.168.2.5     subnet-C
192.168.3.237   subnet-C
192.168.6.188   subnet-D
192.168.7.209   subnet-D
192.168.9.10    subnet-E
# add some /28, /16 case
192.168.12.39   subnet-F
192.168.12.58   subnet-G
10.63.11.1      subnet-I
10.63.102.69    subnet-I

I use netaddr to calculate CIDR, here is my code:

from netaddr import IPAddress, IPNetwork

def netaddr_test(ips, netsets):
    for ip in ips:
        for subnet, name in netsets.iteritems():
            if IPAddress(ip) in IPNetwork(subnet):
                print ip, '\t',  name
                break

netaddr_test(ip_list, netsets)

But this code is too too too slow, it iterate too much. the complexity of time is O(n**2).

Once we have tens of thousands of ip to iterate, this code cost too much time.

Is there any better way to solve this problem?

Answer 1

I can recommend use specially optimized intervaltree module for making search fast. Thus the task could be solved for O(m*log n) time. For example:

   from intervaltree import Interval, IntervalTree
   from ipaddress import ip_network, ip_address

   # build nets tree
   netstree = IntervalTree(
                           Interval(
                                    ip_network(net).network_address, 
                                    ip_network(net).broadcast_address, 
                                    name
                                   ) 
                          for 
                          net, name 
                          in 
                          netsets.items()
                         )

   # Now you may check ip intervals     
   for i in ip_list:
       ip = ip_address(i)
       nets = netstree[ip]
       if nets:   # set is not empty
            netdata = list(nets)[0]
            print(netdata.data)
            # prints 'subnet-E'

Answer 2

# ip address list
ip_list = [
'192.168.1.151', '192.168.10.191', '192.168.6.127',
'192.168.2.227', '192.168.2.5', '192.168.3.237',
'192.168.6.188', '192.168.7.209', '192.168.9.10'
]

# subnet dict
netsets = {
'192.168.1.0/24': 'subnet-A',     # {subnet: subnet's name} 
'192.168.10.0/24': 'subnet-B',
'192.168.2.0/24': 'subnet-C',
'192.168.3.0/24': 'subnet-C',
'192.168.6.0/24': 'subnet-D',
'192.168.7.0/24': 'subnet-D',
'192.168.9.0/24': 'subnet-E',
}
new_netsets = {}
for k,v in netsets.items():
   new_netsets['.'.join(k.split('.')[:3])] = v

for IP in ip_list:
   newIP = '.'.join(IP.split('.')[:3])
   print IP, new_netsets[newIP]

Hope this helps.

Answer 3

I would suggest to avoid creating new instances in the for loop. This will not decrease complexity (it will increase it) but it will speed up the netaddr_test , especially if it is called more than one times. Example:

def _init(ips, netsets):
    """Initialize all objects"""
    new_ips = []
    new_subs = {}
    for ip in ips:
         new_ips.append(IPAddress(ip))

    for subnet, info in netsets.iteritems():

        new_subs[subnet] = {'name': info, 'subnet': IPNetwork(subnet)}

    return new_ips, new_subs

def netaddr_test(ips, netsets):
    for ip in ips:
        for stringnet, info in netsets.iteritems():
            if ip in info['subnet']:
                print ip, '\t',  info['name']
                break

ni, ns = _init(ip_list, netsets)
netaddr_test(ni, ns)

UPDATE: Tested the code above with

ip_list = [
    '192.168.1.151', '192.168.10.191', '192.168.6.127', 
    '192.168.2.227', '192.168.2.5', '192.168.3.237', 
    '192.168.6.188', '192.168.7.209', '192.168.9.10'
] * 1000

Results:

# Original
$ time python /tmp/test.py > /dev/null

real    0m0.357s
user    0m0.345s
sys     0m0.012s

# Modified
$ time python /tmp/test2.py > /dev/null

real    0m0.126s
user    0m0.122s
sys     0m0.005s

Now, I have never used netaddr so I am not sure about how it handles subnets internally. In your case you can see the subnet as a range of IPs and each IP is a uint_32 so you can convert everything to integers:

 # IPs now are 
 ip_list_int = [3232235927, 3232238271, ...]

 netsets_expanded = {
     '192.168.1.0/24': {'name': 'subnet-A', 'start': 3232235776, 'end': 3232236031}

netaddr can be used to convert you data in the above format. Once there, your netaddr_test becomes (and works only with integer comparisons):

def netaddr_test(ips, netsets):
    for ip in ips:
        for subnet, subinfo in netsets.iteritems():
            if ip >= subinfo['start'] and ip < subinfo['end']:
                print ip, '\t',  subinfo.name
                break

Answer 4

In general case where you have N templates and M values to test for match you can do nothing better than O(N*M). But if you can reformulate the task than you can speed it up.

My suggestion is to group templates so that you have a few uplevel templates and if an IP matches it than you go down to final templates. In your examples this would be

grouped_netsets = {
    "192.168.0.0/16":  {
        '192.168.1.0/24': 'subnet-A',     # {subnet: subnet's name} 
        '192.168.10.0/24': 'subnet-B', 
        '192.168.2.0/24': 'subnet-C', 
        '192.168.3.0/24': 'subnet-C',
        '192.168.6.0/24': 'subnet-D', 
        '192.168.7.0/24': 'subnet-D', 
        '192.168.9.0/24': 'subnet-E',
        }
    }   

def netaddr_test(ips, grouped_netsets):
    for ip in ips:
        for group, netsets in grouped_netsets.iteritems():
            if IPAddress(ip) in IPNetwork(group):
                for subnet, name in netsets.iteritems():
                    if IPAddress(ip) in IPNetwork(subnet):
                        print(ip, '\t',  name)
                        break

So if ip_list would contain anything not starting with 192.168 you'll drop it with one check.

The only question remaining is to write the function for grouping the netsets with optimal configuraton.

Answer 5

I mock a ip list and a subnet dict as input:

 # ip address list ip_list = [ '192.168.1.151', '192.168.10.191', '192.168.6.127', '192.168.2.227', '192.168.2.5', '192.168.3.237', '192.168.6.188', '192.168.7.209', '192.168.9.10' ] # subnet dict netsets = { '192.168.1.0/24': 'subnet-A', # {subnet: subnet's name} '192.168.10.0/24': 'subnet-B', '192.168.2.0/24':'subnet-C', '192.168.3.0/24': 'subnet-C', '192.168.6.0/24': 'subnet-D', '192.168.7.0/24': 'subnet-D', '192.168.9.0/24':'subnet-E', } 

and then each ip address in ip_list need to find the name of subnet.

We assume that each ip address can find the corresponding subnet in netsets.

Ouput like this:

 192.168.1.151 subnet-A 192.168.10.191 subnet-B 192.168.6.127 subnet-D 192.168.2.227 subnet-C 192.168.2.5 subnet-C 192.168.3.237 subnet-C 192.168.6.188 subnet-D 192.168.7.209 subnet-D 192.168.9.10 subnet-E 

[...] Is there any better to solve this problem?

Here's a two liner that does it:

for ip_addr in ip_list:
    print "{0}\t{1}".format(ip_addr,netsets[".".join(ip_addr.split('.')[0:-1])+".0/24"])

Answer 6

Assuming that subnets don't overlap each other you could convert the subnet to two integers, beginning and end of the range. These numbers would be added to a list which would be sorted. While doing this we'd need to build a dictionary which could be used later to retrieve the subnet name with start of the range.

def to_int(ip):
    parts = map(int, ip.split('.'))

    return parts[0] << 24 | parts[1] << 16 | parts[2] << 8 | parts[3]

def build(netsets):
    ranges = []
    subnets = {}

    for net, name in netsets.iteritems():
        ip, size = net.split('/')
        start = to_int(ip)
        end = start | 0xffffffff >> int(size)
        ranges.extend([start, end])
        subnets[start] = name

    ranges.sort()
return ranges, subnets

When searching for an IP you'd turn it to number again and do bisect_left on the list or ranges. If result is uneven number or the IP matches on any number on the list then the IP is within a subnet. Then you'd use the star of the range to get the name of the subnet from a dictionary that was built earlier:

def find(ranges, subnets, ip):
    num = to_int(ip)
    pos = bisect.bisect_left(ranges, to_int(ip))

    # Check if first IP in the range
    if pos % 2 == 0 and ranges[pos] == num:
        pos += 1

    if pos % 2:
        return subnets[ranges[pos - 1]]
    else:
        return None

With the previous building block one could easily get the subnet for each IP with following code:

ranges, subnets = build(netsets)
for ip in ip_list:
    print 'ip: {0}, subnet: {1}'.format(ip, find(ranges, subnets, ip))

Building the dictionary and the range list would take O(m log m) time and going through the IP list would take O(n log m) where m is the number of subnets and n the number of IPs. Solution works with different subnets of different size and will print None in case the IP doesn't belong to any subnet.