How to build a Flink application with Maven in Linux

Question

I'm using Maven to manage my custom Flink application in a CentOS server. And I'm a beginner of Java.

To build an empty project, I just followed the official doc , which means that I executed the command below:

mvn archetype:generate                               \
      -DarchetypeGroupId=org.apache.flink              \
      -DarchetypeArtifactId=flink-quickstart-java      \
      -DarchetypeVersion=1.10.0

It works. I get a project as below:

.
├── pom.xml
├── src
    └── main
        ├── java
        │   └── myflink
        │       ├── BatchJob.java
        │       └── StreamingJob.java
        └── resources
            └── log4j.properties

After that, I modify the file StreamingJob.java with the simple example WordCount .

Then I execute mvn clean package to try to build my jar file.

As my understanding, I should get two jar files, one is for the BatchJob.java , the other is for the StreamingJob.java (which is the example of WordCount ).

However, after executing the command mvn clean package , I get myflink-1.jar and original-myflink-1.jar . I can't figure out what they are. Furthermore, it doesn't seem that the BatchJob.java and the StreamingJob.java have their own jar files.

So what is the file myflink-1.jar ? Does it contain the two java files? If I want to get a jar file, which contains only the StreamingJob.java , what should I do?

Answer 1

Your jar file is for your entire package which typically aggregate java class files, some metadata, resource files. The reason it is jar is because of <packaging>jar</packaging> defined in pom.xml. If you expand your jar file you would see both the java class files in respective package.

The reason we see 2 jar file is due to the addition of Maven Shade Plugin. The first jar ( flint-demo-1.0-SNAPSHOT.jar ) is generated from the default maven jar plugin and the other one( original-flint-demo-1.0-SNAPSHOT.jar ) from the maven shade plugin.

Maven Shade plugin helps us generate uber-jar/ fat jar. We can also control the dependency ie include/exclude dependency as done in this sample flink project. So when you run mvn clean package you should see some dependency are being excluded as defined in pom.xml file as shown below

[INFO] --- maven-shade-plugin:3.1.1:shade (default) @ flint-demo ---
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.15 from the shaded jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.7 from the shaded jar.
[INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar.
[INFO] Replacing original artifact with shaded artifact.

To know more about the maven shade plugin you can check here and here

If you want to exclude any file ie in your case say BatchJob you can use Maven Compiler Plugin. I see in sample project maven compiler plugin is already present so you need to just include the <excludes> , something like below

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.1</version>
    <configuration>
        <source>${java.version}</source>
        <target>${java.version}</target>
        <excludes>
            <exclude>com/flint/demo/BatchJob.java</exclude>
        </excludes>
    </configuration>
</plugin>

Now when you do mvn clean package and check the jar file you should not see BatchJob class file.

$ jar tf flint-demo-1.0-SNAPSHOT.jar
META-INF/MANIFEST.MF
META-INF/
com/
com/flint/
com/flint/demo/
com/flint/demo/StreamingJob.class
log4j.properties
META-INF/maven/
META-INF/maven/com.flint.demo/
META-INF/maven/com.flint.demo/flint-demo/
META-INF/maven/com.flint.demo/flint-demo/pom.xml
META-INF/maven/com.flint.demo/flint-demo/pom.properties

To know more about Maven Compiler plugin check here