Best practices for deploying Java webapps with minimal downtime?

Question

When deploying a large Java webapp (>100 MB .war) I\'m currently use the following deployment process:

• The application .war file is expanded locally on the develop

Answer 1

You're using Resin, Resin has built in support for web app versioning.

http://www.caucho.com/resin-4.0/admin/deploy.xtp#VersioningandGracefulUpgrades

Update: It's watchdog process can help with permgenspace issues too.

Answer 2

I'm not sure if this answers your question, but I'll just share on the deployment process I use or encounter in the few projects I did.

Similiar to you, I do not ever recall making a full war redeployment or update. Most of the time, my updates are restricted to a few jsp files, maybe a library, some class files. I am able to manage and determine which are the affected artifacts, and usually, we packaged those update in a zip file, along with an update script. I will run the update script. The script does the following:

• Backup the files that will be overwritten, maybe to a folder with today's date and time.
• Unpackage my files
• Stop the application server
• Move the files over
• Start the application server

If downtime is a concern, and they usually are, my projects are usually HA, even if they are not sharing state but using a router that provide sticky session routing.

Another thing that I am curious would be, why the need to rsync? You should able to know what are the required changes, by determining them on your staging/development environment, not performing delta checks with live. In most cases, you would have to tune your rsync to ignore files anyway, like certain property files that define resources a production server use, like database connection, smtp server, etc.

I hope this is helpful.

Answer 3

I wrote a bash script that takes a few parameters and rsyncs the file between servers. Speeds up rsync transfer a lot for larger archives:

https://gist.github.com/3985742

Answer 4

If static files are a big part of your big WAR (100Mo is pretty big), then putting them outside the WAR and deploying them on a web server (e.g. Apache) in front of your application server might speed up things. On top of that, Apache usually does a better job at serving static files than a servlet engine does (even if most of them made significant progress in that area).

So, instead of producing a big fat WAR, put it on diet and produce:

• a big fat ZIP with static files for Apache
• a less fat WAR for the servlet engine.

Optionally, go further in the process of making the WAR thinner: if possible, deploy Grails and other JARs that don't change frequently (which is likely the case of most of them) at the application server level.

If you succeed in producing a lighter WAR, I wouldn't bother of rsyncing directories rather than archives.

Strengths of this approach:

1. The static files can be hot "deployed" on Apache (e.g. use a symbolic link pointing on the current directory, unzip the new files, update the symlink and voilà).
2. The WAR will be thinner and it will take less time to deploy it.

Weakness of this approach:

1. There is one more server (the web server) so this add (a bit) more complexity.
2. You'll need to change the build scripts (not a big deal IMO).
3. You'll need to change the rsync logic.

Answer 5

As for the early context restarts. All containers have configuration options to disable auto-redeploy on class file or static resource changes. You probably can't disable auto redeploys on web.xml changes so this file is the last one to update. So if you disable to auto redeploy and update the web.xml as the last one you'll see the context restart after the whole update.

Answer 6

We upload the new version of the webapp to a separate directory, then either move to swap it out with the running one, or use symlinks. For example, we have a symlink in the tomcat webapps directory named "myapp", which points to the current webapp named "myapp-1.23". We upload the new webapp to "myapp-1.24". When all is ready, stop the server, remove the symlink and make a new one pointing to the new version, then start the server again.

We disable auto-reload on production servers for performance, but even so, having files within the webapp changing in a non-atomic manner can cause issues, as static files or even JSP pages could change in ways that cause broken links or worse.

In practice, the webapps are actually located on a shared storage device, so clustered, load-balanced, and failover servers all have the same code available.

The main drawback for your situation is that the upload will take longer, since your method allows rsync to only transfer modified or added files. You could copy the old webapp folder to the new one first, and rsync to that, if it makes a significant difference, and if it's really an issue.