How can I start a remote service using Terraform provisioning?

Question

I want my Terraform config to provision a server and start the service at the end by invoking a command and keep running it. I tried using nohup and screen using remote-exec:

Answer 1

Try adding a sleep after your nohup. Worked for me. I suspect backgrounding your last remote-exec lets Terraform get away with shutting down the connection before the child process has a chance to start up, despite the nohup.

provisioner "remote-exec" {
    inline = [
        "nohup sudo command &",
        "sleep 1"
    ]
}

Answer 2

A more robust solution to this is to start a service that runs your process instead.

This means that the init system can take control of the process and restart it if necessary. It also gains the other benefits of a modern init system such as handling dependency ordering (making sure that other services are running before that one starts) and things like logs.

If you set your service to start on boot then you can avoid having to connect to the server over SSH as well and can mean that the server will tolerate reboots without needing to be reprovisioned.

With Systemd this would mean creating a unit file that could be as simple as the following:

[Unit]
Description=foo

[Service]
ExecStart=command
Restart=always

[Install]
WantedBy=multi-user.target

Running the following commands would then make sure that command is ran at boot automatically and that any failures of the process would lead to it being automatically restarted:

systemctl enable foo.service
systemctl start foo.service

This becomes even more important when using mechanisms such as AWS' autoscaling groups to provision your instances. When creating autoscaling groups via the aws_autoscaling_group resource you are unable to easily connect to the instances created at that time and have no control of connecting to instances as the group scales out or replaces instances. At this point it's important that the instance is able to configure itself entirely either from the base image alone (which could be created using a tool such as Packer) or through user data scripts that are automatically ran on first boot.