How to handle 3rd party JARs and remote repository in Maven POM?

Question

I'm using a Maven dependency that requires setting up a remote repository. In the same project, I'm using a custom built JAR and trying to add it as a dependency as well. The problem is that I get errors saying that Maven cannot find my custom JAR in the remote repository.

In my POM I have multiple dependencies, including my custom built dependency and the dependency requiring the remote repository (confluent). I have tried putting my custom dependency first in the POM and that did not help. I tried removing the repository from the POM and I don't get the error about my custom built dependency, but I get an error for the remote one.

I'm running the code in a Maven Docker container. I've tried running the Docker container with a Bash shell and without the Maven commands, then manually ran the Maven commands inside the container, and manually checked the ~/.m2/repository and confirmed that my custom built JAR is in there.

Ran an interactive Maven container:

docker run -it --rm --name ProcessedObsGen -v "$(pwd)":/usr/src/mymaven \
  -w /usr/src/mymaven maven:3.3-jdk-8 /bin/bash

Inside the Docker container:

mvn clean install:install-file \
  -Dfile=/usr/src/mymaven/libs/daas-utilities-0.0.1-SNAPSHOT.jar \
  -DgroupId=atlas -DartifactId=daas-utilities -Dversion=0.0.1-SNAPSHOT -Dpackaging=jar
mvn exec:java -Dexec.mainClass="atlas.processed_obs_generator.App"
ls ~/.m2/repository

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>atlas</groupId>
    <artifactId>processed-obs-generator</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>processed-obs-generator</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>0.10.2.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>1.9.0</version>
        </dependency>
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
            <version>1.7.25</version>
        </dependency>
        <dependency>
            <groupId>atlas.daas-utilities</groupId>
            <artifactId>daas-utilities</artifactId>
            <version>0.0.1-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-avro-serializer</artifactId>
            <version>5.0.0</version>
        </dependency>
    </dependencies>
    <repositories>
        <repository>
            <id>confluent</id>
            <url>https://packages.confluent.io/maven/</url>
        </repository>
    </repositories>
</project>

I run the code as a Docker container with the command:

docker run -it --rm --name ProcessedObsGen -v "$(pwd)":/usr/src/mymaven \
  -w /usr/src/mymaven maven:3.3-jdk-8 mvn clean install:install-file \
  -Dfile=/usr/src/mymaven/libs/daas-utilities-0.0.1-SNAPSHOT.jar \
  -DgroupId=atlas -DartifactId=daas-utilities -Dversion=0.0.1-SNAPSHOT -Dpackaging=jar; \
  mvn exec:java -Dexec.mainClass="atlas.processed_obs_generator.App"

I get the error:

Failed to execute goal on project processed-obs-generator: Could not resolve dependencies for project atlas:processed-obs-generator:jar:0.0.1-SNAPSHOT: Could not find artifact atlas.daas-utilities:daas-utilities:jar:0.0.1-SNAPSHOT in confluent ( https://packages.confluent.io/maven/ )

I've also tried downloading the confluent jar, putting it in the same location that I'm putting my custom jar, installing it the same way, and removing the remote repository from my pom. Then I get errors with the confluent classes I'm using saying ClassNotFoundException.

Answer 1

I was able to solve it.

I rebuilt my utilities jar with dependencies included. Please see Including dependencies in a jar with Maven .

I kept my pom pretty much the same, although I removed some dependencies not being used.

<dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>atlas</groupId>
            <artifactId>daas-utilities</artifactId>
            <version>0.0.1-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>io.confluent</groupId>
            <artifactId>kafka-avro-serializer</artifactId>
            <version>5.0.0</version>
        </dependency>
    </dependencies>
    <repositories>
        <repository>
            <id>confluent</id>
            <url>https://packages.confluent.io/maven/</url>
        </repository>
    </repositories>

I placed my custom built jar in ${project.basedir}/libs.

I adjusted the maven commands in my docker run command to include my jar with dependencies. I also figured out that the mvn install:install-file ... is only installing the jar and is not (as I assumed) also installing your project. So, I added another mvn clean install command at the end before I executed it.

docker run -it --rm --name ProcessedObsGen -v "$(pwd)":/usr/src/mymaven -w /usr/src/mymaven maven:3.3-jdk-8 mvn clean install:install-file -Dfile=/usr/src/mymaven/libs/daas-utilities-0.0.1-SNAPSHOT-jar-with-dependencies.jar -DgroupId=atlas -DartifactId=daas-utilities -Dversion=0.0.1-SNAPSHOT -Dpackaging=jar; mvn clean install; mvn exec:java -Dexec.mainClass="atlas.processed_obs_generator.App"

Answer 2

In your install-file commands you define -DgroupId=atlas while your POM reads:

       ...
       <dependency>
            <groupId>atlas.daas-utilities</groupId>
            ...