Maven build [WARNING] we have a duplicate class

Question

Anybody has any idea what happened to my maven build? I am getting a lot of duplicate warnings.

[WARNING] We have a duplicate org/apache/commons/logging/impl         


        

Answer 1

In my case I was relying on a package that also creates a shaded jar.

Shaded jars are meant for deployment, not installing as a dependency.

Creating a reduced dependency POM during the build process of the dependency, instructs maven on which dependencies can be left out.

In the maven-shade-plugin configuration:

<configuration>
  <createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>

For more details see this post:

What is the maven-shade-plugin used for, and why would you want to relocate java packages?

The error I was getting from maven:

WARNING: x.jar, y.jar contain overlapping classes

Answer 2

Take a look at the "Dependency Exclusions" section in the Maven doc.

In your provided example, I'll exclude the commons-logging:commons-logging-api:jar:1.0.4:compile dependency from org.apache.hadoop.hive:hive-common:jar:0.7.1-cdh3u3:compile. In your pom.xml :

    <dependency>
        <groupId>org.apache.hadoop.hive</groupId>
        <artifactId>hive-common:jar</artifactId>
        <version>0.7.1-cdh3u3</version>
        <exclusions>
            <exclusion>
                <groupId>commons-logging</groupId>
                <artifactId>commons-logging-api</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

Answer 3

You can exclude the jar you don't want (the ones that are giving the duplicate warnings using the following tags under the shade plugin -

    <configuration>
    <artifactSet>
      <excludes>
        <exclude>commons-logging:commons-logging</exclude>
      </excludes>
    </artifactSet>
    <minimizeJar>true</minimizeJar>
    </configuration>

More details can be found at http://maven.apache.org/plugins/maven-shade-plugin/shade-mojo.html

Answer 4

I saw this happen in eclipse when I updated my parent project's dependencies.

I deleted all the files in my target directory and it fixed the issues.

Answer 5

In my case, my parent pom was including commons-beanutils and my child module (which is the only thing I wanted to compile) was including commons-io.

The shade plug in complained about duplicates since commons-io and commons-beansutil shared some common classes. Note that beansutiul was being included even though it was not needed, and was not used.

I solve this by minimizing the jar by adding this to the configuration:

<minimizeJar>true</minimizeJar>

Now the shade plugin did not add unused resources.

Warning went away.

Answer 6

All above (about reviewing dependencies tree and excluding) is correct in the most of cases, but in my case (i didn't have overlapping in my dependencies) preliminary clean helped (don't know why though):

mvn clean package