AWS/EBS Background Jobs in Rails

Question

I have an app on AWS using Elastic Beanstalk/RDS/EC2/etc written in Rails.

The app has some database entries that need to be deleted after a certain time, and was thinking of using a gem to create a background task.

However, all the gems I see are just ways of programmatically creating cron tasks. With Elastic Beanstalk, the server instance being used will scale/change so cron tasks will be lost. Is there another way to set up a background job within Rails?

Answer 1

I typically solve this problem by using container_commands that you can configure in a .config file inside .ebextensions folder in your app. The idea is that the code is executed on so-called 'leader' instance in your webservers pool. It is not considered as a bulletproof solution but does the job and mostly works (it will fail when autoscaling comes into play though). Use distributed Mutex lock (redis-mutex or database backed) to block all processes except one that obtains lock.

#create config file, use numbers/names that suit your needs:
#.ebextensions/02_container_commands.config 
container_commands:
  01_cron_tasks:
    command: "cat .ebextensions/cron_tasks.txt > /etc/cron.d/yourapp && chmod 644 /etc/cron.d/yourapp"
    leader_only: true

cron will pick this, just put the crontab definition in .ebextensions/cron_tasks.txt file and add also this one to your Git repo.

0 5 * * * root bash -l -c "su -m webapp; cd /var/app/current && rake your_rake_task"

It's sad but it seems that Amazon AWS team still did not come up with a proper, convenient solution for scheduling background jobs with Rails in Elastic Beanstalk environments. Their Worker Tier requires that you add a dedicated endpoint in your app that listens to messages from their own cron implementation. This is bad because it forces you to add a hosting-specific code to your core application logic (routes, controllers) and requires more servers running.

Another disadvantage in AWS when comparing to platforms like Heroku is the lack of one-click support for Redis-backed queueing systems like Sidekiq and on-demand workers that could be spawned via cron-like schedulers.