Managing Beanstalkd Queues Per User

Question

Let's say for arguments sake, I have a single SMTP server where I can send 60 emails per minute (1 per second).

If I have 60 users wanting to send emails through this SMTP server and they all send me 60 emails.

If I maintain 1 queue in Laravel, all 60 emails from user #1 will get sent first, then all 60 emails from user #2 will get sent next and so on. User #60 will have to wait for 3,540 emails to send before his starts.

I would like use Beanstalkd on a round-robin basis, and the only way I can think of doing this, is maintaining 60 tubes, one for each user but this would require me to run:

php artisan queue:listen --queue=user1,user2,user3,etc,user60

The problem I am facing here is that if user #61 signs up, his tube will not be picked up by the listener until I create a new command like so:

php artisan queue:listen --queue=user1,user2,user3,etc,user60,user61

If I grow to 1 million users, is it advisable to have 1 million tubes?

Is there any easy way to ask Beanstalkd to fetch 1 job per tube because these tubes can be created on the fly, I will never know how many tubes I have?

The problem I am facing is that I have a bottleneck on the SMTP server so I need to send these emails in a round robin fashion. I can achieve this easily using MySQL and Cron Jobs but this has led to duplicate emails in the past when using MySQL as a queueing system.

Answer 1

Short answer, you can try adding more workers, but if you have a funnel on the SMTP server, you only need to remember that you are replacing one problem with another.

Client 1 has 60 emails
Client 2 has 60 emails and so on ... until client 60

Client 1 will only send the second email one minute later, thus it will take one hour to send 60 emails from client 1.

You would be better served using a service such as Mandrill or Sendgrid, where you would have multiple servers sending the messages.