How do you remotely update Java applications?

Question

We\'ve got a Java server application that runs on a number of computers, all connected to the Internet, some behind firewalls. We need to remotely update the JAR files and start

Answer 1

We use Eclipse which the update system of OSGi, and our experience is very good.

Recommended!

Answer 2

It's very hard to make the update atomic, especially if you have any database updating to do.

But, if you don't, what you can do is first make sure your application can run from a relative path. That is, you can put your app in some directory, and all of the important files are found relative to that location, so that that your actual installation location is not really important.

Next, duplicate your installation. Now, you have the "running" version and you have the "new" version.

Update the "new" version using whatever tech you like (FTP, rsync, paper tape, whatever floats your boat).

Verify your installation (checksums, quick unit tests, whatever you need -- even start it up on a test port if you like).

When you are happy with the new installation, down the original running instance, RENAME the original directory (mv application application_old), rename the NEW directory (mv application_new application), and start it back up.

Your down time is reduced to server shut down and start up time (since rename is "free").

If, by chance, you detect a critical error, you have your original version still there. Stop the new server, rename it back, restart the old. Very fast fall back.

The other nice thing is that your service infrastructure is static (like your rc scripts, cron jobs, etc.) since they point to the "application" directory, and it doesn't change.

It can also be done with soft links instead of renaming directories. either way is fine.

But the technique is simple, and near bullet proof if your application cooperates.

Now, if you have DB changes, well, that's completely different nasty issue. Ideally if you can make your DB changes "backward compatible", then hopefully the old application version can run on the new schema, but that's not always possible.

Answer 3

The latest version of Java Web Start allows for injecting an application in the local cache without actually invoking the program and it can be marked as "offline". Since the cache is what is used to invoke the program, it will be updated for the next run only. For this to work you will most likely need jars with version numbers in their name (e.g. our-library-2009-06-01.jar).

Answer 4

I would use ansible to distribute jars to multiple servers and run update scripts.

For an update to be unnoticeable for users the application would have to stop the old instance and bring up a new one very quickly, in which java apps aren't very good.

There are ways to make update without any downtime, but none of them is free. Choosing a solution you should consider what downtime is acceptable to you and what cost can you accept to achieve it.

I see two options:

1. Rolling update, which means each time you update you spin up an instance with the new version, then validate if it is working correctly, and if it is, you kill the old version. This solution will require some kind of proxy (eg. haproxy) which will direct the traffic to the valid instances.
2. Optimizing application start-up time.

As Will Hartung mentioned you will also have to consider external dependencies like database schema which you might also want to update. In order to get seamless updates, you might also want to do a rolling update on a database. It would require you to do not introduce any breaking changes between any two consecutive releases of the application. Eg. when you want to delete a column, in the first release you remove all references to the column in an application and in next release you are allowed to delete the column in the database.

Answer 5

You should definitely take a look at OSGi, it was created just for these cases (especially for embedded products) and is used by a large number of companies. You can update jar "bundles", add and remove them, while the app is running. I haven't used it myself, so I don't know about the quality of the open source frameworks/servers, but here is a bunch of useful links to get you started:

http://www.osgi.org/Main/HomePage
http://www.aqute.biz/Code/Bnd
http://blog.springsource.com/2008/02/18/creating-osgi-bundles/
http://blog.springsource.com/
http://www.knopflerfish.org/
http://felix.apache.org/site/index.html

Answer 6

You didn't specify the type of server apps - I'm going to assume that you aren't running web apps (as deploying a WAR already does what you are talking about, and you very rarely need a web app to do pull type updates. If you are talking about a web app, the following discussion can still apply - you'll just implement the update check and ping-pong for the WAR file instead of individual files).

You may want to take a look at jnlp - WebStart is based on this (this is a client application deployment technology), but I'm pretty sure that it could be tailored to performing updates for a server type app. Regardless, jnlp does a pretty good job of providing descriptors that can be used for downloading required versions of required JARs...

Some general thoughts on this (we have several apps in the same bucket, and are considering an auto-update mechanism):

1. Consider having a bootstrap.jar file that is capable of reading a jnlp file and downloading required/updated jars prior to launching the application.

2. JAR files can be updated even while an app is running (at least on Windows, and that is the OS most likely to hold locks on running files). You can run into problems if you are using custom class loaders, or you have a bunch of JARs that might be loaded or unloaded at any time, but if you create mechanisms to prevent this, then overwriting JARs then re-launching the app should be sufficient for update.

3. Even though it is possible to overwrite JARs, you might want to consider a ping-pong approach for your lib path (if you don't already have your app launcher configured to auto-read all jar files in the lib folder and add them to the class path automatically, then that's something you really do want to do). Here's how ping-pong works:

App launches and looks at lib-ping\version.properties and lib-pong\version.properties and determines which is newer. Let's say that lib-ping has a later version. The launcher searches for lib-ping*.jar and adds those files to the CP during the launch. When you do an update, you download jar files into lib-pong (or copy jar files from lib-ping if you want to save bandwidth and the JAR didn't actually change - this is rarely worth the effort, though!). Once you have all JARs copied into lib-pong, the very last thing you do is create the version.properties file (that way an interrupted update that results in a partial lib folder can be detected and purged). Finally, you re-launch the app, and bootstrap picks up that lib-pong is the desired classpath.

4. ping-pong as described above allows for a roll-back. If you design it properly, you can have one piece of your app that you test the heck out of and then never change that checks to see if it should roll-back a given version. That way if you do mess up and deploy something that breaks the app, you can invalidate the version. This part of the application just has to delete the version.properties file from the bad lib-* folder, then re-launch. It's important to keep this part dirt simple because it's your fail safe.

5. You can have more than 2 folders (instead of ping/pong, just have lib-yyyymmdd and purge all but the newest 5, for example). This allows for more advanced (but more complicated!) rollback of JARs.