实现BlackList的最有效方法

Question

I developing a Ip filter and was guessing how i could, using any type of esque data structure, develop a VERY efficient and fast BlackList filter.

What i want to do is simple, every incoming/outcoming connection i have to check in a list of blocked IP´s.

The IPs are scattered, and the memory use should be linear(not dependent of the number of blocked list, because i want to use on limited systems(homebrew routers)).

I have time and could create anything from zero. The difficulty is not important to me. If you can use anything, what you should do ?

Answer 1

Hashtables are the way to go. They have averaged O(1) complexity for lookup, insertion and deletion! They tend to occupy more memory than trees but are much faster.

Since you are just working with 32 bit integer (you can of course convert an IP to a 32 bit integer) things will be amazingly simple and fast.

You can just use a sorted array. Insertion and removal cost is O(n) but lookup is O(log n) and especially memory is just 4 byte for each ip. The implementation is very simple, perhaps too much :D

Binary trees have complexity of O(log n) for lookup, insertion and deletion. A simple binary tree would not be sufficient however, you need an AVL tree or a Red Black Tree, that can be very annoying and complicated to implement. AVL and RBT trees are able to balance themselves, and we need that because an unbalanced tree will have a worst time complexity of O(n) for lookup, that is the same of a simple linked list!

If instead of single and unique ip u need to ban ip ranges, probably you need a Patricia Trie, also called Radix Tree, they were invented for word dictionaries and for ip dictionaries. However these trees can be slower if not well written\\balanced. Hashtable are always better for simple lookups! They are too fast to be real :)

Now about synchronization:

If you are filling the black list only once at application startup, you can use a plain read only hashtable (or radix tree) that don't have problems about multithreading and locking.

If you need to update it not very often, I would suggest you the use reader-writer locks.

If you need very frequent updates I would suggest you to use a concurrent hashtable. Warning: don't write your own, they are very complicated and bug prone, find an implementation on the web! They use a lot the (relatively) new atomic CAS operations of new processors (CAS means Compare and Swap). These are a special set of instructions or sequence of instructions that allow 32 bit or 64 bit fields on memory to be compared and swapped in a single atomic operation without the need of locking. Using them can be complicated because you have to know very well your processor, your operative system, your compiler and the algorithm itself is counterintuitive. See http://en.wikipedia.org/wiki/Compare-and-swap for more informations about CAS.

Concurrent AVL tree was invented, but it is so complicated that I really don't know what to say about these :) for example, http://hal.inria.fr/docs/00/07/39/31/PDF/RR-2761.pdf

I just found that concurrent radix tree exists: ftp://82.96.64.7/pub/linux/kernel/people/npiggin/patches/lockless/2.6.16-rc5/radix-intro.pdf but it is quite complicated too.

Concurrent sorted arrays doesn't exists of course, you need a reader-writer lock for update.

Consider also that the amount of memory required to handle a non-concurrent hashtable can be quite little: For each IP you need 4 byte for the IP and a pointer. You need also a big array of pointers (or 32 bit integers with some tricks) which size should be a prime number greater than the number of items that should be stored. Hashtables can of course also resize themselves when required if you want, but they can store also more item than that prime numbers, at the cost of slower lookup time.

For both trees and hashtable, the space complexity is linear.

I hope this is a multithreading application and not a multiprocess application (fork). If it is not multithreading you cannot share a portion of memory in a fast and reliable way.

Answer 2

The "most efficient" is a hard term to quantify. Clearly, if you had unlimited memory, you would have a bin for every IP address and could immediately index into it.

A common tradeoff is using a B-tree type data structure. First level bins could be preset for the first 8 bits of the IP address, which could store a pointer to and the size of a list containing all currently blocked IP addresses. This second list would be padded to prevent unnecessary memmove() calls and possibly sorted. (Having the size and the length of the list in memory allows an in-place binary search on the list at the slight expensive of insertion time.)

For example:

127.0.0.1 =insert=> { 127 :: 1 }
127.0.1.0 =insert=> { 127 :: 1, 256 }
12.0.2.30 =insert=> { 12 : 542; 127 :: 1, 256 }

The overhead on such a data structure is minimal, and the total storage size is fixed. The worse case, clearly, would be a large number of IP addresses with the same highest order bits.

Answer 3

One way to improve the performance of such a system is to use a Bloom Filter. This is a probabilistic data structure, taking up very little memory, in which false positives are possible but false negatives are not.

When you want to look up an IP address, you first check in the Bloom Filter. If there's a miss, you can allow the traffic right away. If there's a hit, you need to check your authoritative data structure (eg a hash table or prefix tree).

You could also create a small cache of "hits in the Bloom Filter but actually allowed" addresses, that is checked after the Bloom Filter but before the authoritative data structure.

Basically the idea is to speed up the fast path (IP address allowed) at the expense of the slow path (IP address denied).