What if Redis queue doesn't fit in memory?

Question

I would like to use Redis for job queue purposes. What I want to learn is what happens if my queue doesn't fit in memory and how it affects performance?

I don't have very much resources for this purpose; only server with 8GB memory and also MySQL, Apache and several web sites with medium load are running on this server. Should I restrict memory usage of Redis?

My job queue will include approximately 100 million ID of associated rows in MySQL server. ID's are auto incremental integers.

I will move ID's from MySQL to Redis at first and then I'll use this queue for several jobs and I'll delete all processed IDs.

Now, for this scenario, do you recommend usage of Redis as queue manager or something else, and how?

Thank you for your ideas :)

Answer 1

What I want to learn is what happens if my queue doesn't fit in memory and how it affects performance?

According to FAQ if the server runs out of RAM it will start swapping and redis performance will degrade. If you set maxmemory option in config then the write operations will reply with errors. You can alternatively look into custom sharding .

Should I restrict memory usage of Redis?

First you should try to estimate memory usage in order to figure out if your RAM will be enough. Depending on your data structure it can fit into memory and maybe your wouldn't have to deal with sharding or performance degradation. Another approach to lower the memory usage is to optimize .