哪个更快？ 是Memcache还是文件查询？ （使用maxmind geoip.dat文件）

Question

I'm using Python on Appengine and am looking up the geolocation of an IP address like this:

import pygeoip
gi = pygeoip.GeoIP('GeoIP.dat')
Location = gi.country_code_by_addr(self.request.remote_addr)

(pygeoip can be found here: http://code.google.com/p/pygeoip/ )

I want to geolocate each page of my app for a user so currently I lookup the IP address once then store it in memcache.

My question - which is quicker? Looking up the IP address each time from the .dat file or fetching it from memcache? Are there any other pros/cons I need to be aware of?

For general queries like this, is there a good guide to teach me how to optimise my code and run speed tests myself? I'm new to python and coding in general so apologies if this is a basic concept.

Thanks!

Tom

EDIT: Thanks for the responses, memcache seems to be the right answer. I think that Nick and Lennart are suggesting that I add the whole gi variable to memcache. I think this is possible. FYI - the whole GeoIP.dat file is just over 1MB so not that large.

Answer 1

What takes time there is rather loading the database from the dat file. Once you have that in memory, the lookup time is not significant. So if you can keep the gi variable in memory that seems the best solution.

If you can't you probably can't use memcached either.

Answer 2

If you need to do lookups across multiple processes (which you almost certainly do on AppEngine), and you are likely to encounter the same ip address lots of times in a short time span (which you probably are), then using memcache is probably a good idea for speed.

More details, since you said you were relatively new to coding:

As Lennart Regebro correctly says, the slow thing is reading the geoip file from disk and parsing it. Individual queries will then be fast. However, if any given process is only serving one request (which, from your perspective, on AppEngine, it is), then this price will get paid on each request. Caching recently used lookups in memcache will let you share this information across processes...but only for recently encountered data points. However, since any given ip is likely to show up in bursts (because it is one user interacting with your site), this is exactly what you want.

Other alternatives are to pre-load all the data points into memcache. You probably don't want to do this, since you have a limited amount of memory available, and you won't end up using most of it. (Also, memcache will throw parts of it away if you hit your memory limit, which means you'd need to write backup code to read from the geoip database live anyway.) In general, doing lazy caching -- look up a value the slow way when you first need it and then keep it around for re-use -- is a very effective mechanism. Memcache is specifically geared for this, since it throws away data that hasn't been used recently when it encounters memory pressure.

Another alternative in general (although not in AppEngine) is to run a separate process that handles just location queries, and having all your front-end processes talk to it (eg via thrift). Then you could use the suggestion of just loading up the geoip database in that process and querying it live for each request.

Hope that helps some.

Answer 3

For individual IP addresses that you already have gotten out of the database, I would put them in memcache for sure. I am assuming the database file is relatively large, and you don't want to load that from memcache every time you need to look up one address.

One tool I know people use to help track speed of API calls is AppStats . It can help you see how long various calls to the APIs are taking.

Since you are new to programming in general, I will mention that appstats is a very App Engine specific tool. If you were just writing a basic python application that was going to run on your own computer, you could do timing of things by simply subtracting two timestamps:

import time
t1 = time.time()
#do whatever it is you want to time here.
t2 = time.time()
elapsed_time = t2-t1