Trigger Heroku worker

Question

What are ways to trigger a worker process in Heroku? In particular, a process that is needed infrequently but quickly when needed, eg bluemoon.js .

Polling every second to read a task queue (which can be stored in a database) is the approach I can think of.

Trigger makes more sense to me for this case. Is there a way to directly trigger a worker process when needed? Or is there no real downside to frequent polling?

Answer 1

What you really want is a message queueing service like Amazon SQS , RabbitMQ , or something similar.

What message queueing services do is this:

• You have your web dyno fire off a message into a messaging service that says "Hey! Run this task. Here's some data to process."
• The message service then grabs this message, and relays it (quickly) to any of your worker dynos.
• Your worker dynos then complete the work that needs to be done, and can communicate back to the messaging service that the job has been finished.

The reason the above pattern works so well is that these services are optimized for speed and cost -- they're VERY inexpensive to run (I'm a huge fan of Amazon SQS myself), have almost no overhead, and are incredibly fast.

The reason you DON'T want to poll a database (which is what most people think of when they imagine stuff like this) is because it's going to waste resources and cause problems later on:

• You'll be constantly hitting your database server from your worker dynos, using a lot of unnecessary bandwidth / IO / CPU resources.
• You'll be constantly hitting your database server making queries, which is going to slow down your database and reduce the amount of important queries it can run.

In general, for problems like this, a messaging service is the perfect solution!