I am using forever js to keep my node server running 24/7 on AWS EC2.
I use this command
forever start index.js
However, I notice that
So this is an example of using cron
to run scripts that can restart service/perform some automated task. Basically, I created some scripts that I need to run at certain time intervals on my server. For your case, you want to make a script that will automatically check the state of your forever.js and if it returns a bad response, run the forever restartall
command that you mention above.
You can set this up by creating a new crontab entry on the server. As far as the script goes, I'm by no means a bash script guru; I made a simple script that works for me. Here is an example of checking a service on my machine, restarting it if it is not running.
#!/bin/bash
zabbix_server="service zabbix-server"
zabbix_agent="service zabbix-agent"
logfile=zabbix_auto_restart.log
logfilePath=/etc/scripts/zabbix/$logfile
zabbix_server_running=0
zabbix_agent_running=0
grep_agent (){
local retval=$(ps -ef | grep -v grep | grep zabbix_agentd | wc -l)
echo $retval
}
grep_server (){
local retval=$(ps -ef | grep -v grep | grep zabbix_server | wc -l)
echo $retval
}
check_zabbix_agentd (){
if (( $(grep_agent) <= 0 ))
then
sudo /etc/init.d/zabbix-agent start
echo `date` "$zabbix_agent was stopped... Restarting" >> $logfilePath
echo "************************************************" >> $logfilePath
#Send email to notify that the script ran
echo "$(date) $zabbix_agent was restarted from zabbix_restart.sh" | mutt -s "Zabbix Auto-restart Script Just Ran" <my-email>
else
let zabbix_agent_running=1
fi
}
check_zabbix_server (){
if (( $(grep_server) <= 0 ))
then
sudo /etc/init.d/zabbix-server start
echo `date` "$zabbix_server was stopped... Restarting" >> $logfilePath
echo "************************************************" >> $logfilePath
#Send email to notify that the script ran
echo "$(date) $zabbix_server was restarted from zabbix_restart.sh" | mutt -s "Zabbix Auto-restart Script Just Ran" evan.bechtol@ericsson.com
else
let zabbix_server_running=1
fi
}
main_loop (){
until ((zabbix_server_running == 1 && zabbix_agent_running == 1));
do
check_zabbix_agentd
check_zabbix_server
sleep 1.5
done
}
main_loop
Have you ever considered pm2 as an alternative. pm2 has some pretty neat features like:
You can control restarts also by adding a --restart-delay <delay>
option.
Hereby a full output of the help of this tool.
pm2 --help
-------------
Looking for a complete monitoring and management tool for PM2?
_ _ _ _
| | _____ _ _ _ __ ___ ___| |_ _ __(_) ___ ___ (_) ___
| |/ / _ \ | | | '_ ` _ \ / _ \ __| '__| |/ __/ __| | |/ _ \
| < __/ |_| | | | | | | __/ |_| | | | (__\__ \_| | (_) |
|_|\_\___|\__, |_| |_| |_|\___|\__|_| |_|\___|___(_)_|\___/
|___/
Features
- Real Time Dashboard
- CPU/Memory monitoring
- HTTP monitoring
- Event notification
- Custom value monitoring
- Real Time log display
Checkout
https://keymetrics.io/
-------------
[PM2] Spawning PM2 daemon
[PM2] PM2 Successfully daemonized
Usage: pm2 [cmd] app
Commands:
start [options] <file|json|stdin|app_name|pm_id...> start and daemonize an app
deploy <file|environment> deploy your json
startOrRestart <json> start or restart JSON file
startOrReload <json> start or gracefully reload JSON file
startOrGracefulReload <json> start or gracefully reload JSON file
stop [options] <id|name|all|json|stdin...> stop a process (to start it again, do pm2 restart <app>)
restart [options] <id|name|all|json|stdin...> restart a process
scale <app_name> <number> scale up/down a process in cluster mode depending on total_number param
reload <name|all> reload processes (note that its for app using HTTP/HTTPS)
gracefulReload <name|all> gracefully reload a process. Send a "shutdown" message to close all connections.
id <name> get process id by name
delete <name|id|script|all|json|stdin...> stop and delete a process from pm2 process list
sendSignal <signal> <pm2_id|name> send a system signal to the target process
ping ping pm2 daemon - if not up it will launch it
updatePM2 update in-memory PM2 with local PM2
update (alias) update in-memory PM2 with local PM2
install|module:install <module|git:/> install or update a module and run it forever
module:update <module|git:/> update a module and run it forever
module:generate [app_name] Generate a sample module in current folder
uninstall|module:uninstall <module> stop and uninstall a module
publish|module:publish Publish the module you are currently on
set <key> <value> sets the specified config <key> <value>
multiset <value> multiset eg "key1 val1 key2 val2
get [key] get value for <key>
conf [key] [value] get / set module config values
config <key> [value] get / set module config values
unset <key> clears the specified config <key>
interact [options] [secret_key|command] [public_key] [machine_name] linking action to keymetrics.io - command can be stop|info|delete|restart
link [options] [secret_key|command] [public_key] [machine_name] linking action to keymetrics.io - command can be stop|info|delete|restart
web launch a health API on port 9615
dump dump all processes for resurrecting them later
save (alias) dump all processes for resurrecting them later
resurrect resurrect previously dumped processes
startup [platform] auto resurrect process at startup. [platform] = ubuntu, centos, redhat, gentoo, systemd, darwin, amazon
logrotate copy default logrotate configuration
generate generate an ecosystem.json configuration file
ecosystem generate an ecosystem.json configuration file
reset <name|id|all> reset counters for process
describe <id> describe all parameters of a process id
desc <id> (alias) describe all parameters of a process id
info <id> (alias) describe all parameters of a process id
show <id> (alias) describe all parameters of a process id
list list all processes
ls (alias) list all processes
l (alias) list all processes
status (alias) list all processes
jlist list all processes in JSON format
prettylist print json in a prettified JSON
monit launch termcaps monitoring
m (alias) launch termcaps monitoring
flush flush logs
reloadLogs reload all logs
logs [options] [id|name] stream logs file. Default stream all logs
kill kill daemon
pull <name> [commit_id] updates repository for a given app
forward <name> updates repository to the next commit for a given app
backward <name> downgrades repository to the previous commit for a given app
gc force PM2 to trigger garbage collection
deepUpdate performs a deep update of PM2
*
Options:
-h, --help output usage information
-V, --version output the version number
-v --version get version
-s --silent hide all messages
-m --mini-list display a compacted list without formatting
-f --force force actions
-n --name <name> set a <name> for script
-i --instances <number> launch [number] instances (for networked app)(load balanced)
-l --log [path] specify entire log file (error and out are both included)
-o --output <path> specify out log file
-e --error <path> specify error log file
-p --pid <pid> specify pid file
-k --kill-timeout <delay> delay before sending final SIGKILL signal to process
--max-memory-restart <memory> specify max memory amount used to autorestart (in megaoctets)
--restart-delay <delay> specify a delay between restarts (in milliseconds)
--env <environment_name> specify environment to get specific env variables (for JSON declaration)
-x --execute-command execute a program using fork system
-u --user <username> define user when generating startup script
--hp <home path> define home path when generating startup script
-c --cron <cron_pattern> restart a running process based on a cron pattern
-w --write write configuration in local folder
--interpreter <interpreter> the interpreter pm2 should use for executing app (bash, python...)
--interpreter-args <arguments> interpret options (alias of --node-args)
--log-date-format <momentjs format> add custom prefix timestamp to logs
--no-daemon run pm2 daemon in the foreground if it doesn't exist already
--skip-env do not refresh environmnent on restart/reload
--source-map-support force source map support
--only <application-name> with json declaration, allow to only act on one application
--disable-source-map-support force source map support
--merge-logs merge logs from different instances but keep error and out separated
--watch [paths] watch application folder for changes
--ignore-watch <folders|files> folder/files to be ignored watching, chould be a specific name or regex - e.g. --ignore-watch="test node_modules "some scripts""
--node-args <node_args> space delimited arguments to pass to node in cluster mode - e.g. --node-args="--debug=7001 --trace-deprecation"
--no-color skip colors
--no-vizion start an app without vizion feature (versioning control)
--no-autorestart start an app without automatic restart
--no-treekill Only kill the main process, not detached children
--no-pmx start an app without pmx
--no-automation start an app without pmx
Basic Examples:
Start an app using all CPUs available + set a name :
$ pm2 start app.js -i 0 --name "api"
Restart the previous app launched, by name :
$ pm2 restart api
Stop the app :
$ pm2 stop api
Restart the app that is stopped :
$ pm2 restart api
Remove the app from the process list :
$ pm2 delete api
Kill daemon pm2 :
$ pm2 kill
Update pm2 :
$ npm install pm2@latest -g ; pm2 update
More examples in https://github.com/Unitech/pm2#usagefeatures
Deployment help:
$ pm2 deploy help
This video is part of an awesome node-deployment-series ending up with using a service on a UNIX-server... Maybe it helps you. By the way, it's worth watching all four videos... Youtube-Link
Figured it out. The problem was due to less RAM available in my EC2 instance. I upgraded RAM to 2 GB and have not experienced the crash so far.
Node can also make uncaught exception errors but my major problem was due to less memory.
Hope this helps others.
I'd suggest the usage of PM2
This is the best option to run on a production server.
What are the advantages of running your application this way?
/home/safeuser/.pm2/logs/app-err.log
.ref: https://www.digitalocean.com/community/tutorials/how-to-use-pm2-to-setup-a-node-js-production-environment-on-an-ubuntu-vps
A (NodeJS) server should not stop for no reason. Most of the time, it's because of a 500 Error that have not been catch
ed and stop the server, then you will have to restart it.
forever
is using node
by default to start your server.
nodemon
is a npm package that restart your server when the code changes or when your server stops.
You can use forever
and nodemon
together by doing :
forever start nodemon --exitcrash app.js
or
forever start -c nodemon app.js
Or, as suggested in other answers, you can use PM2, which would be better for production !