User-data scripts is not running on my custom AMI, but working in standard Amazon linux

Question

I searched a lot of topic about \"user-data script is not working\" in these few days, but until now, I haven\'t gotten any idea about my case yet, please help me to figure

Answer 1

On ubuntu 16, removing /var/lib/cloud/* does not work. I removed only instance(s) from the folder /var/lib/cloud/ and then it ran fine for me

I ran:

sudo rm /var/lib/cloud/instance
sudo rm /var/lib/cloud/instances

Then I retried my user data script and it worked fine

Answer 2

User Data should execute fine without using #cloud-boothook (which is used to activate the User Data at the earliest possible time during the boot process).

I started a new Amazon Linux AMI and used your User Data, plus a bit extra:

#!/bin/bash

echo 'bar' > /tmp/bar
echo 'test' > /home/ec2-user/user-script-output.txt
echo 'foo' > /tmp/foo

This successfully created three files.

User Data scripts are executed as root, so it should have permission to create files in any location.

I notice that in your supplied code, one example refers to /home/ec2-user/user-script/output.txt (with a subdirectory) and one example refers to /home/ec2-user/user-script-output.txt (no subdirectory). The command would understandably fail if you attempt to create a file in a non-existent directory, but your "Update" example seems to show that it did actually work.

Answer 3

I was having a lot of trouble with this. I'll provide detailed walk-though.

My added wrinkle is that I'm using terraform to instantiate the hosts via a launch configuration and autoscaling group.

I could NOT get it to work by adding the script inline in lc.tf

user_data                   = DATA <<
"
#cloud-boothook
#!/bin/bash
echo 'some crap'\'
"
DATA

I could fetch it from user data,

wget http://169.254.169.254/latest/user-data

but noticed I was getting it with the quotes still in it.

This is how I got it to work: I moved to pulling it from a template instead since what you see is what you get.

user_data                   = "${data.template_file.bootscript.rendered}"

This means I also need to declare my template file like so:

data "template_file" "bootscript" {
  template = "${file("bootscript.tpl")}"

}

But I was still getting an error in the cloud init logs /var/log/cloud-init.log [WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'Content-Type: text/cloud...'

Then I found this article about user data formatting user That makes sense, if user-data can come in multiple parts, maybe cloud-init needs cloud-init commands in one place and the script in the other.

So my bootscript.tpl looks like this:

Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0

--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"

#cloud-config
cloud_final_modules:
- [scripts-user, always]

--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"

#!/bin/bash
echo "some crap"
--//

Answer 4

I'm using CentOS and the logic for userdata there is simple:

• In the file /etc/rc.local there is a call for a initial.sh script, but it looks for a flag first:

  if [ -f /var/tmp/initial ]; then
      /var/tmp/initial.sh &
  fi
  

initial.sh is the file that does the execution of user-data, but in the end it deletes the flag. So, if you want your new AMI to execute user-data again, just create the flag again before create the image:

touch /var/tmp/initial

Answer 5

I have also faced the same issue on Ubuntu 16.04 hvm AMI. I have raised the issue to AWS support but still I couldn't find exact reason/bug which affects it.

But still I have something which might help you.

Before taking AMI remove /var/lib/cloud directory (each time). Then while creating Image, set it to no-reboot.

If these things still ain't working, you can test it further by forcing user-data to run manually. Also tailf /var/log/cloud-init-output.log for cloud-init status. It should end with something like modules:final to make your user-data run. It should not stuck on modules:config.

sudo rm -rf /var/lib/cloud/* sudo cloud-init init sudo cloud-init modules -m final

I don't have much idea whether above commands will work on CentOS or not. I have tested it on Ubuntu.

In my case, I have also tried removing /var/lib/cloud directory, but still it failed to execute user-data in our scenario. But I have came up with different solution for it. What we have did is we have created script with above commands and made that script to run while system boots.

I have added below line in /etc/rc.local to make it happen.

sudo bash /home/ubuntu/force-user-data.sh || exit 1

But here is the catch, it will execute the script on each boot so which will make your user-data to run on every single boot, just like #cloud-boothook. No worries, you can just tweak it by just removing the force-user-data.sh itself at the end. So your force-user-data.sh will look something like

#!/bin/bash sudo rm -rf /var/lib/cloud/* sudo cloud-init init sudo cloud-init modules -m final sudo rm -f /home/ubuntu/force-user-data.sh exit 0

I will appreciate if someone can put some lights on why it is unable to execute the user-data.

Answer 6

Also, If we are using user interactive commands like

sudo yum install java-1.8.0-devel

Then, we need to use with flags like -y

sudo yum install java-1.8.0-devel -y

You can find this in the EC2 documentation under Run commands at launch.

Ref: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts