Node JS auto restart all forever JS process when server goes down / crashes

前端 未结 6 914
眼角桃花
眼角桃花 2021-02-07 05:37

I am using forever js to keep my node server running 24/7 on AWS EC2.

I use this command

forever start index.js

However, I notice that

相关标签:
6条回答
  • 2021-02-07 06:05

    So this is an example of using cronto run scripts that can restart service/perform some automated task. Basically, I created some scripts that I need to run at certain time intervals on my server. For your case, you want to make a script that will automatically check the state of your forever.js and if it returns a bad response, run the forever restartall command that you mention above.

    You can set this up by creating a new crontab entry on the server. As far as the script goes, I'm by no means a bash script guru; I made a simple script that works for me. Here is an example of checking a service on my machine, restarting it if it is not running.

    #!/bin/bash
    zabbix_server="service zabbix-server"
    zabbix_agent="service zabbix-agent"
    logfile=zabbix_auto_restart.log
    logfilePath=/etc/scripts/zabbix/$logfile
    zabbix_server_running=0
    zabbix_agent_running=0
    
    grep_agent (){
            local retval=$(ps -ef | grep -v grep | grep zabbix_agentd | wc -l)
            echo $retval
    }
    
    grep_server (){
            local retval=$(ps -ef | grep -v grep | grep zabbix_server | wc -l)
            echo $retval
    }
    
    check_zabbix_agentd (){
            if (( $(grep_agent) <= 0 ))
            then
               sudo /etc/init.d/zabbix-agent start
               echo `date` "$zabbix_agent was stopped... Restarting" >> $logfilePath
               echo "************************************************" >> $logfilePath
    
               #Send email to notify that the script ran
               echo "$(date) $zabbix_agent was restarted from zabbix_restart.sh" | mutt -s "Zabbix Auto-restart Script Just Ran" <my-email>
    
            else
               let zabbix_agent_running=1
            fi
    }
    
    check_zabbix_server (){
            if (( $(grep_server) <= 0 ))
            then
               sudo /etc/init.d/zabbix-server start
               echo `date` "$zabbix_server was stopped... Restarting" >> $logfilePath
               echo "************************************************" >> $logfilePath
    
               #Send email to notify that the script ran
               echo "$(date) $zabbix_server was restarted from zabbix_restart.sh" | mutt -s "Zabbix Auto-restart Script Just Ran" evan.bechtol@ericsson.com
    
            else
               let zabbix_server_running=1
            fi
    }
    
    main_loop (){
            until ((zabbix_server_running == 1 && zabbix_agent_running == 1));
            do
                    check_zabbix_agentd
                    check_zabbix_server
                    sleep 1.5
            done
    }
    
    main_loop
    
    0 讨论(0)
  • 2021-02-07 06:10

    Have you ever considered pm2 as an alternative. pm2 has some pretty neat features like:

    • Running a cluster
    • Restarting instances in the cluster one by one (zero downtime deployments)
    • Resource monitoring (shows realtime cpu and memory usage)
    • Manage all pm2 processes using the cli

    You can control restarts also by adding a --restart-delay <delay> option.

    Hereby a full output of the help of this tool.

    pm2 --help
    
                        -------------
    Looking for a complete monitoring and management tool for PM2?
     _                             _        _            _
    | | _____ _   _ _ __ ___   ___| |_ _ __(_) ___ ___  (_) ___
    | |/ / _ \ | | | '_ ` _ \ / _ \ __| '__| |/ __/ __| | |/ _ \
    |   <  __/ |_| | | | | | |  __/ |_| |  | | (__\__ \_| | (_) |
    |_|\_\___|\__, |_| |_| |_|\___|\__|_|  |_|\___|___(_)_|\___/
         |___/
    
                          Features
    
                   - Real Time Dashboard
                   - CPU/Memory monitoring
                   - HTTP monitoring
                   - Event notification
                   - Custom value monitoring
                   - Real Time log display
    
                          Checkout
    
                   https://keymetrics.io/
    
                        -------------
    
    [PM2] Spawning PM2 daemon
    [PM2] PM2 Successfully daemonized
    
    Usage: pm2 [cmd] app
    
    Commands:
    
    start [options] <file|json|stdin|app_name|pm_id...>                  start and daemonize an app
    deploy <file|environment>                                            deploy your json
    startOrRestart <json>                                                start or restart JSON file
    startOrReload <json>                                                 start or gracefully reload JSON file
    startOrGracefulReload <json>                                         start or gracefully reload JSON file
    stop [options] <id|name|all|json|stdin...>                           stop a process (to start it again, do pm2 restart <app>)
    restart [options] <id|name|all|json|stdin...>                        restart a process
    scale <app_name> <number>                                            scale up/down a process in cluster mode depending on total_number param
    reload <name|all>                                                    reload processes (note that its for app using HTTP/HTTPS)
    gracefulReload <name|all>                                            gracefully reload a process. Send a "shutdown" message to close all connections.
    id <name>                                                            get process id by name
    delete <name|id|script|all|json|stdin...>                            stop and delete a process from pm2 process list
    sendSignal <signal> <pm2_id|name>                                    send a system signal to the target process
    ping                                                                 ping pm2 daemon - if not up it will launch it
    updatePM2                                                            update in-memory PM2 with local PM2
    update                                                               (alias) update in-memory PM2 with local PM2
    install|module:install <module|git:/>                                install or update a module and run it forever
    module:update <module|git:/>                                         update a module and run it forever
    module:generate [app_name]                                           Generate a sample module in current folder
    uninstall|module:uninstall <module>                                  stop and uninstall a module
    publish|module:publish                                               Publish the module you are currently on
    set <key> <value>                                                    sets the specified config <key> <value>
    multiset <value>                                                     multiset eg "key1 val1 key2 val2
    get [key]                                                            get value for <key>
    conf [key] [value]                                                   get / set module config values
    config <key> [value]                                                 get / set module config values
    unset <key>                                                          clears the specified config <key>
    interact [options] [secret_key|command] [public_key] [machine_name]  linking action to keymetrics.io - command can be stop|info|delete|restart
    link [options] [secret_key|command] [public_key] [machine_name]      linking action to keymetrics.io - command can be stop|info|delete|restart
    web                                                                  launch a health API on port 9615
    dump                                                                 dump all processes for resurrecting them later
    save                                                                 (alias) dump all processes for resurrecting them later
    resurrect                                                            resurrect previously dumped processes
    startup [platform]                                                   auto resurrect process at startup. [platform] = ubuntu, centos, redhat, gentoo, systemd, darwin, amazon
    logrotate                                                            copy default logrotate configuration
    generate                                                             generate an ecosystem.json configuration file
    ecosystem                                                            generate an ecosystem.json configuration file
    reset <name|id|all>                                                  reset counters for process
    describe <id>                                                        describe all parameters of a process id
    desc <id>                                                            (alias) describe all parameters of a process id
    info <id>                                                            (alias) describe all parameters of a process id
    show <id>                                                            (alias) describe all parameters of a process id
    list                                                                 list all processes
    ls                                                                   (alias) list all processes
    l                                                                    (alias) list all processes
    status                                                               (alias) list all processes
    jlist                                                                list all processes in JSON format
    prettylist                                                           print json in a prettified JSON
    monit                                                                launch termcaps monitoring
    m                                                                    (alias) launch termcaps monitoring
    flush                                                                flush logs
    reloadLogs                                                           reload all logs
    logs [options] [id|name]                                             stream logs file. Default stream all logs
    kill                                                                 kill daemon
    pull <name> [commit_id]                                              updates repository for a given app
    forward <name>                                                       updates repository to the next commit for a given app
    backward <name>                                                      downgrades repository to the previous commit for a given app
    gc                                                                   force PM2 to trigger garbage collection
    deepUpdate                                                           performs a deep update of PM2
    *
    
    Options:
    
    -h, --help                           output usage information
    -V, --version                        output the version number
    -v --version                         get version
    -s --silent                          hide all messages
    -m --mini-list                       display a compacted list without formatting
    -f --force                           force actions
    -n --name <name>                     set a <name> for script
    -i --instances <number>              launch [number] instances (for networked app)(load balanced)
    -l --log [path]                      specify entire log file (error and out are both included)
    -o --output <path>                   specify out log file
    -e --error <path>                    specify error log file
    -p --pid <pid>                       specify pid file
    -k --kill-timeout <delay>            delay before sending final SIGKILL signal to process
    --max-memory-restart <memory>        specify max memory amount used to autorestart (in megaoctets)
    --restart-delay <delay>              specify a delay between restarts (in milliseconds)
    --env <environment_name>             specify environment to get specific env variables (for JSON declaration)
    -x --execute-command                 execute a program using fork system
    -u --user <username>                 define user when generating startup script
    --hp <home path>                     define home path when generating startup script
    -c --cron <cron_pattern>             restart a running process based on a cron pattern
    -w --write                           write configuration in local folder
    --interpreter <interpreter>          the interpreter pm2 should use for executing app (bash, python...)
    --interpreter-args <arguments>       interpret options (alias of --node-args)
    --log-date-format <momentjs format>  add custom prefix timestamp to logs
    --no-daemon                          run pm2 daemon in the foreground if it doesn't exist already
    --skip-env                           do not refresh environmnent on restart/reload
    --source-map-support                 force source map support
    --only <application-name>            with json declaration, allow to only act on one application
    --disable-source-map-support         force source map support
    --merge-logs                         merge logs from different instances but keep error and out separated
    --watch [paths]                      watch application folder for changes
    --ignore-watch <folders|files>       folder/files to be ignored watching, chould be a specific name or regex - e.g. --ignore-watch="test node_modules "some scripts""
    --node-args <node_args>              space delimited arguments to pass to node in cluster mode - e.g. --node-args="--debug=7001 --trace-deprecation"
    --no-color                           skip colors
    --no-vizion                          start an app without vizion feature (versioning control)
    --no-autorestart                     start an app without automatic restart
    --no-treekill                        Only kill the main process, not detached children
    --no-pmx                             start an app without pmx
    --no-automation                      start an app without pmx
    
    Basic Examples:
    
    Start an app using all CPUs available + set a name :
    $ pm2 start app.js -i 0 --name "api"
    
    Restart the previous app launched, by name :
    $ pm2 restart api
    
    Stop the app :
    $ pm2 stop api
    
    Restart the app that is stopped :
    $ pm2 restart api
    
    Remove the app from the process list :
    $ pm2 delete api
    
    Kill daemon pm2 :
    $ pm2 kill
    
    Update pm2 :
    $ npm install pm2@latest -g ; pm2 update
    
    More examples in https://github.com/Unitech/pm2#usagefeatures
    
    Deployment help:
    
    $ pm2 deploy help
    
    0 讨论(0)
  • 2021-02-07 06:12

    This video is part of an awesome node-deployment-series ending up with using a service on a UNIX-server... Maybe it helps you. By the way, it's worth watching all four videos... Youtube-Link

    0 讨论(0)
  • 2021-02-07 06:12

    Figured it out. The problem was due to less RAM available in my EC2 instance. I upgraded RAM to 2 GB and have not experienced the crash so far.

    Node can also make uncaught exception errors but my major problem was due to less memory.

    Hope this helps others.

    0 讨论(0)
  • 2021-02-07 06:24

    I'd suggest the usage of PM2

    This is the best option to run on a production server.

    What are the advantages of running your application this way?

    • It's easy to setup and run.
    • PM2 will automatically restart your application if it crashes.
    • PM2 will keep a log of your unhandled exceptions - in this case, in a file at /home/safeuser/.pm2/logs/app-err.log.
    • With one command, PM2 can ensure that any applications it manages restart when the server reboots. Basically meaning your node application will start as a service.

    ref: https://www.digitalocean.com/community/tutorials/how-to-use-pm2-to-setup-a-node-js-production-environment-on-an-ubuntu-vps

    0 讨论(0)
  • 2021-02-07 06:28

    A (NodeJS) server should not stop for no reason. Most of the time, it's because of a 500 Error that have not been catched and stop the server, then you will have to restart it. forever is using node by default to start your server.

    nodemon is a npm package that restart your server when the code changes or when your server stops.

    You can use forever and nodemon together by doing :

    forever start nodemon --exitcrash app.js
    

    or

    forever start -c nodemon app.js
    

    Or, as suggested in other answers, you can use PM2, which would be better for production !

    0 讨论(0)
提交回复
热议问题