With huge thanks to this post by Daniel Viklund, I want to share some notes and code about how I am managing delayed_job on a Rails 3.2.16 app deployed on AWS Elastic Beanstalk.
I’m currently using the 64bit Amazon Linux 2014.09 v1.1.0 running Ruby 2.0 (Passenger Standalone). The “2014.09” is significant as that generation of the Beanstalk AMIs was the first to eliminate certain environment variables, which means that scripts now need to declare them explicitly. (More info in this AWS forum thread and this one.)
delayed_job Logging
The main difference in my approach compared to Daniel’s is that I wanted to direct the delayed_job log to the same folder where Beanstalk stores Passenger logs, /var/app/support/logs. This directory path is accessible from a Beanstalk AMI by running the command:
/opt/elasticbeanstalk/bin/get-config container -k app_log_dir
To be consistent with Passenger, the log should be owned by the webapp user.
Script to Restart delayed_job after Loading App
Save the following in your .ebextensions folder, e.g. as delayed-job.config:
# Adapted from http://www.dannemanne.com/posts/post-deployment_script_on_elastic_beanstalk_restart_delayed_job
commands:
create-post-dir:
command: "mkdir /opt/elasticbeanstalk/hooks/appdeploy/post"
ignoreErrors: true
files:
"/opt/elasticbeanstalk/hooks/appdeploy/post/restart_delayed_job.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
# Loading environment data
EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
EB_APP_DEPLOY_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
EB_APP_PID_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)
EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)
# Export EB_APP_LOG_DIR so we can access it when running script/delayed_job below,
# which accesses config/initializers/delayed_job.rb, which uses EB_APP_LOG_DIR.
export EB_APP_LOG_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_log_dir)
# Make sure the delayed_job.log exists and is owned by $EB_APP_USER
touch $EB_APP_LOG_DIR/delayed_job.log
chown $EB_APP_USER:$EB_APP_USER $EB_APP_LOG_DIR/delayed_job.log
# Setting up correct environment and ruby version so that bundle can load all gems
. $EB_SUPPORT_DIR/envvars
. $EB_SCRIPT_DIR/use-app-ruby.sh
# Now we can do the actual restart of the worker. Make sure to have double quotes when using env vars in the command.
# For Rails 4, replace script/delayed_job with bin/delayed_job
cd $EB_APP_DEPLOY_DIR
su -s /bin/bash -c "bundle exec script/delayed_job --pid-dir=$EB_APP_PID_DIR restart" $EB_APP_USER
su -s /bin/bash -c "bundle exec script/delayed_job --pid-dir=$EB_APP_PID_DIR status" $EB_APP_USER
In my tests, scripts in the /opt/elasticbeanstalk/hooks/appdeploy/post directory are executed when you deploy a new version of your code and whenever the EC2 instance is rebooted. However this behavior is not even documented, much less guaranteed to continue, so you’ll need to check this as AMI versions are updated.
delayed_job Configuration
Here is my config/initializers/delayed_job.rb file. I’ve overridden a few defaults and set up logging.
require 'delayed/command'
# Adapted from https://github.com/collectiveidea/delayed_job
Delayed::Worker.destroy_failed_jobs = false # default true
# Delayed::Worker.sleep_delay = 60 # polling frequency; default 5 seconds
Delayed::Worker.max_attempts = 3 # default 25
Delayed::Worker.max_run_time = 5.minutes # default 4.hours
# Delayed::Worker.read_ahead = 5 # default 5
# Delayed::Worker.default_queue_name = 'default' # default: process without named queue
# Delayed::Worker.delay_jobs = !Rails.env.test? # default true - queue (delay) even in Test environment
Rails.logger.debug ">> #{Time.now}: delayed_job.rb running as user (whoami): #{%x(whoami)}"
Rails.logger.debug ">> #{Time.now}: delayed_job.rb Delayed::Worker.logger at proc start: #{Delayed::Worker.logger}"
# Set the logger if it isn't set yet. Actually this file seems to execute every time we start a Rails
# instance or console, the logger is never set beforehand, but leave the test here anyway.
if Delayed::Worker.logger.nil?
# .ebextensions/80-delayed-job.config exports the EB_APP_LOG_DIR environment variable so we can use it here.
# If EB_APP_LOG_DIR is not found, use "standard" application log path.
eb_app_log_dir = ENV['EB_APP_LOG_DIR'] || "" # if nil, make it ""
if Dir.exists?(eb_app_log_dir)
log_file = File.join(eb_app_log_dir, 'delayed_job.log')
else
log_file = Rails.root.join('log', 'delayed_job.log')
end
Delayed::Worker.logger = Logger.new(log_file)
Rails.logger.debug ">> #{Time.now}: delayed_job.rb set log path to #{log_file} "
else
# Syntax for getting log path:
# http://jordan.broughs.net/archives/2014/09/provide-separate-rails-log-files-for-each-unicorn-worker
Rails.logger.debug ">> #{Time.now}: delayed_job.rb NOT setting log path - already set to " + \
"#{Delayed::Worker.logger.instance_variable_get(:@logdev).dev.path} "
end
Rails.logger.debug ">> #{Time.now}: delayed_job.rb Delayed::Worker.logger at proc end: #{Delayed::Worker.logger}"
Manually Control delayed_job
As mentioned in the comments, the delayed_job.rb code apparently runs every time you start Rails, even a Rails Console. It also runs if you run the delayed_job script manually, e.g. to check status. If you need to run the delayed_job script from the console, declare the environment $EB_APP_LOG_DIR environment variable first:
sudo su - # Control delayed_job daemon (first setting $EB_APP_LOG_DIR env var) export EB_APP_LOG_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_log_dir) script/delayed_job --pid-dir=/var/app/support/pids status # For stopping/starting, run as webapp: su -s /bin/bash -c "bundle exec script/delayed_job --pid-dir=/var/app/support/pids stop" webapp su -s /bin/bash -c "bundle exec script/delayed_job --pid-dir=/var/app/support/pids start" webapp su -s /bin/bash -c "bundle exec script/delayed_job --pid-dir=/var/app/support/pids restart" webapp
Thanks for such a great write up; this was exactly what I was looking for. I was getting permissions errors on the bin/delayed_job script, so it was failing at first. I just added su -s /bin/bash -c “chmod +x bin/delayed_job” (rails4) right above the last two lines in delayed-job.config and that resolved it, although it could be specific to my situation.