如何正确基准化Python脚本连接到Redis所需的时间？

Question

Recently I was assigned the task to benchmark the time it takes for Python to connect to Redis. The purpose was to see whether there is any point in using connection pools over creating new connections.

I assumed the task to be fairly simple, I tried many cases, however, I'm constantly being told that my tests are completely wrong, so I do not know how to approach this.

Note: I'm relatively new to Python and I haven't gotten the grasp of many things concerning this language, so I don't have much experience.

My first try was the simplest way I could benchmark it that I could think of. This is my try with the script that creates a new connection every time:

#!/usr/bin/env python3

import redis
import timeit
from datetime import datetime

def main():
    connection = redis.StrictRedis(host='localhost', port=6379, db=0)

start_time = datetime.now()
timeit_results = timeit.timeit("main()", setup="from __main__ import main", number=5000)
time_taken = datetime.now() - start_time

print("Timeit results: {}".format(timeit_results))
print("Datetime results: {}".format(time_taken))

And to test the time it would take by using connection pools:

#!/usr/bin/env python3
import redis
import timeit
from datetime import datetime

pool = redis.ConnectionPool(host='localhost', port=6379, db=0)

def main():
    connection = redis.StrictRedis(connection_pool=pool)

start_time = datetime.now()
timeit_results = timeit.timeit("main()", setup="from __main__ import main", number=5000)
time_taken = datetime.now() - start_time

print("Timeit results: {}".format(timeit_results))
print("Datetime results: {}".format(time_taken))

I was told those tests are wrong. So I thought that I'd need to close the connections manually first. However, as I am using StrictRedis , I couldn't find a way (in the documentation) to manually close the connections, so I googled and found that StrictRedis.client_pool.disconnect() would be the solution.

So I wanted to check whether it really works and checked it in the following way:

interface = redis.StrictRedis(host='localhost', port=6379, db=0)

def main():
    connection = redis.StrictRedis(host='localhost', port=6379, db=0)
    connection.connection_pool.disconnect()
    print("Open connections during test: {}".format(len(interface.client_list())))

start_time = datetime.now()
timeit_results = timeit.timeit("main()", setup="from __main__ import main", number=10)
time_taken = datetime.now() - start_time

print("Timeit results: {}".format(timeit_results))
print("Datetime results: {}".format(time_taken))

From which I get the following result:

Open connections during test: 2
Open connections during test: 3
Open connections during test: 4
Open connections during test: 5
Open connections during test: 6
Open connections during test: 7
Open connections during test: 8
Open connections during test: 9
Open connections during test: 10
Open connections during test: 11

Which seemed odd, and to check whether client_list() really returns the correct thing I want it to, I ran a test with connections that can actually be closed (in the main() function):

connection = redis.Connection(host='localhost', port=6379, db=0)
connection.connect()
connection.disconnect()
print("Open connections during test: {}".format(len(interface.client_list())))

Which returns:

Open connections during test: 1
Open connections during test: 1
Open connections during test: 1
Open connections during test: 1
Open connections during test: 1
Open connections during test: 1
Open connections during test: 1
Open connections during test: 1
Open connections during test: 1
Open connections during test: 1

So, it does work. And because StrictRedis.connection_pool.disconnect() was not working, I just thought that StrictRedis connection has no way to be closed, it gets closed automatically.

However, then I was then told that there is a way to close a StrictRedis connection and I have to do that. Later, I was given the hint that I need to avoid using connection pools in the first test, and I saw in the source code that StrictRedis actually creates a connection pool for itself, though I don't really understand it.

Question:
How to correctly benchmark the time it takes for the script to connect to Redis?

Update :

I tried testing the connections via redis.Connection , since it doesn't actually use a connection pool (or at least I couldn't find information about it using one), and also to a redis server that is on another server in the local network:

#!/usr/bin/env python3

import redis
from datetime import datetime
import timeit

test_num = 5000

def timefunc():
    connection = redis.Connection(host='10.**.**.**', port=6379, db=0)
    connection.connect()
    connection.disconnect()


timeit_result = timeit.timeit("timefunc()", setup="from __main__ import timefunc", number=test_num)

print("Timeit result (total): {}".format(timeit_result))
print("Timeit result (average): {}".format(timeit_result / test_num))

Now this was the test that I thought would be correct. However, I am not sure about that, since each time I run the script, totally different results are yielded:

Test 1:
Timeit result (total): 3.4030799390748143
Timeit result (average): 0.0006806159878149628

Test 2:
Timeit result (total): 2.9527969888877124
Timeit result (average): 0.0005905593977775425

Test 3:
Timeit result (total): 6.543300905032083
Timeit result (average): 0.0013086601810064166

Test 4:
Timeit result (total): 21.31216992996633
Timeit result (average): 0.004262433985993266

Test 5:
Timeit result (total): 5.312907374929637
Timeit result (average): 0.0010625814749859273

These tests show really weird results. I restart redis-server each time before I run a test, now I'm on my knees begging for help as I have no idea what am I doing wrong and why is this happening.

Getting back to the original question..
Are those results normal, and why? Is this the correct way I should benchmark the connections, if not, how?

Python version : 3.5
Redis server version : 3.2.6

Thanks in advance.

Answer 1

Speaking about Redis it is usually discussed to keep 1 shared connection or a connection pool. Opening and closing connections every time is dramatically expensive, just don't do this. If you want to open and close connections every time, no need to measure, pool will win big time.

Regarding your test, it is hard to say if you're doing it right, because you're measuring a library - a black box. Who knows what it does under the hood. From here you have two ways: dig into library or measure not connection time, but performance on your app using one approach or another. I'd suggest to measure doing useful operation with pool and without.

But again, throw away an idea of opening and closing connections for every operation.