Storm command fails with NoClassDefFoundError after adding jsoup as provided dependency

Question

I\'m using JSoup in my project and I\'ve declared the dependency in my POM file. It compiles just fine and runs fine too, but only when I used the jar wit

Answer 1

Storm script doesn't use the CLASSPATH variable but instead puts all the jars in the STORM_DIR/lib directory in its classpath. So you have 2 choices:

1. Change the scope of the JSoup dependency to "compile" scope and have it packaged inside the jar with dependencies.
2. Leave the JSoup dependency in "provided" scope and copy the JSoup jar to STORM_DIR/lib directory so that the storm script will automatically put that jar in its classpath.

I would strongly recommend option 1 and follow the standard Maven approach.

Just FYI this is how Storm script creates the classpath string:

def get_classpath(extrajars):
    ret = get_jars_full(STORM_DIR)
    ret.extend(get_jars_full(STORM_DIR + "/lib"))
    ret.extend(extrajars)
    return normclasspath(":".join(ret))

Answer 2

This is as much a question about Maven as it is about Storm and its deployment model. You have to check out what the storm command actually does. First of all, it's actually a Python script that ultimately calls java.

If you look at the function get_classpath(extrajars), you'll note that it does not use the $CLASSPATH evironment variable at all. Rather, it loads the core Storm jars and any jars that you have under a directory lib/ relative to your working directory, as well as config files under ~/.storm

(You will find that ignoring $CLASSPATH is very common in many Java applications. Usually the first thing a "launch script" does is overwrite the CLASSPATH or not use it at all. This is to prevent unknown / unsupported / earlier versions of your jars from causing problems in your application).

As to your application fails when jsoup is declared "provided": when you declare the jar as a provided dependency, it will not be packaged in your "jar with dependencies" assembly. See this question for a good explanation: Difference between maven scope compile and provided for JAR packaging

The tl;dr explanation is that compile scope is shipped with your uber-jar, provided scope isn't, because it's expected to be "provided" by the container you are deploying to. Typically, the "container" is a Java web server, like Tomcat (hence, you should never have to ship JSP or Servlet jars with your web apps in Java). In this case, the "container" that you are expecting to "provide" classes is Storm. However, jsoup is not provided by Storm, hence your error.

Compile-scope classes still need to be shipped with your application because your application will be instantiating / using interfaces, enums, etc.

My recommendation is to just declare jsoup "compile" scope and move on. The alternative will be to write your own bespoke deployment script and/or assembly that puts jsoup under lib/ - essentially the same thing in the end.

Answer 3

Maven scope provided means the dependencies are included at compilation, but not on runtime. The container / run script is expected to include it explicitly, so I can see you're on the right track.

Other things you can check to fix the problem is:

1. Check the CLASSPATH environment variable on the shell instance that runs the java program. Although you already have correct CLASSPATH value on your user shell, often you have to create a new shell instance (ie: running a script) and the CLASSPATH variable is not propagated. On UNIX this is typically done using export command
2. Check the classpath path is valid, has correct filesystem permission, jars are not corrupted
3. Check the java command you used to run the program. If you specify -cp it might (or not) override the CLASSPATH environment variable