AWS Elastic Beanstalk Initializes App Every 24 Hours

My Elastic Beanstalk deployment was loading my app but returning an error:

[Instance: i-4e74d545 Module: AWSEBAutoScalingGroup ConfigSet: null] Command failed on instance. Return code: 1 Output: Error occurred during build: Command hooks failed .

Script /opt/elasticbeanstalk/hooks/appdeploy/post/ 05_revert_bundle_install_local.sh failed with returncode 1

Then I noticed something odd in the logs: the app was trying to reload every 24 hours. One time, my web app actually went offline at a reload (RED status), then 24 hours later, it came back up, without any intervention on my part.

Amazon forum support has suggested that I delete the 05_revert_bundle_install_local.sh script, which is left over from the sample app. I’ll try that, but first I wanted to figure out why it kept trying to reload. Here’s what I found:

1. /var/log/messages shows "Sleeping 86400" coming from [eb-cfn-init].

2. /var/log/eb-cfn-ini.log shows script commands running every day, like this:

+ sleep_delay 
+ (( 163840 < 86400 )) 
+ echo Sleeping 86400 
Sleeping 86400 
+ sleep 86400 
+ true 
+ set +e 
+ cfn_init _AppInstall 
++ EB_EVENT_FILE=/var/log/eb-startupevents.log 
++ EB_SYSTEM_STARTUP=true 
++ /opt/aws/bin/cfn-init -v -s arn:aws:cloudformation:us-west-2:123456789012:stack/awseb-e-3hfcud2zmi-stack/[guid] -r AWSEBAutoScalingGroup --region us-west-2 --configsets _AppInstall 
+ FN_OUTPUT= 
+ RESULT=1 
+ set -e 
+ echo Command Returned: 
Command Returned: 
+ [[ 1 -ne 0 ]] 
+ echo 'Command return code 1' 
Command return code 1 
+ tail_logs 
[log tails omitted]

3. sudo grep -r -H "sleep_delay" /var | cut -d: -f1

led to several files that look like scripts but are stored in text files. For example:

/var/lib/cloud/instances/i-b9c9c0b1/scripts/user-data.txt, but that has a SLEEP_TIME_MAX=3600.

4. sudo grep -r -H "MAX=86400" /var | cut -d: -f1

led back to /var/log/eb-cfn-ini.log. At the top of the file, "+ SLEEP_TIME_MAX=86400" appears to come from a public script downloaded from S3:

https://s3-us-west-2.amazonaws.com/elasticbeanstalk-env-resources-us-west-2/stalks/eb_ruby_passenger_2.8.1/lib/UserDataScript.sh

Here is a copy of the script.

Near the bottom of that script, we have

if [[ "$EB_IS_WORKFLOW_RUNNING" == "true" ]];
then
  APP_CONFIGSET='Hook-PreInit'
else
  echo Worflow not running.  Doing full install.
  APP_CONFIGSET='_AppInstall'
  # Writing startup version to prevent duplicate execution
  retry_execute write_metadata
fi
retry_execute cfn_init $APP_CONFIGSET

The retry_execute function apparently executes the cfn_init function repeatedly until it returns 0. It looks like SLEEP_TIME starts at 10 seconds, then doubles on each execution up to a max of 86400 seconds (one day). In eb-cfn-init.log, I see on 7/4 at 16:19, SLEEP_TIME of 20 seconds is compared against MAX_SLEEP_TIME:


(( 20 < 86400 ))

That keeps doubling to 40, 80, 160, 320 … until it reaches 163840 on 7/6 at 14:28. After that, it just falls back to MAX_SLEEP_TIME of 86400 and executes every day.

5. sudo find / -name 'UserDataScript.sh'

does not find file–maybe it’s not stored on local system, at least not with that name?

I guess the bottom line is, even if your web server seems to be working, if your deployment has any errors (does not return “0”), Beanstalk and its CloudFormation foundation may keep trying to initialize, potentially endangering the stability of your app.

Leave a Reply