How to make Jenkins consider two different builds of a Maven -SNAPSHOT jar artifact identical as part of Continuous Delivery?

Question

EDIT: This is about doing Continuous Delivery with Maven and having it orchestrated with Jenkins. Maven is definitively not designed for that, and this question is part of our effort to get an efficient workflow without using Maven releases. Help is appreciated.

---

We use Maven -SNAPSHOTs within major versions to ensure customers always get the latest code for that given version, which works well. For technical reasons we have two independent Maven jobs - one for compiling sources to jars, and one for combining the appropriate jars to a given deployment. This also works well.

We then have Jenkins orchestrating when to invoke the various steps, and this is where it gets a bit tricky, because if we do the normal mvn clean install in step one, this means that all the snapshot artifacts get recompiled, which in turn makes Jenkins think that all the snapshots changed (as their fingerprint - aka MD5 checksum - changed) even if the sources used to generate the artifacts did not change, triggering all the downstream builds instead of just those which dependencies did change.

I have so far identified these things as varying between builds:

• META-INF/maven/.../pom.properties (as it contains a timestamp)
• META-INF/MANIFEST.MF (contains JDK and user)
• timestamps in jar file

I have found ways around the two first, but the latter is a bit more difficult. It appears that AbstractZipArchiver (which does all the work in zipFile() and zipDir()) is not written to allow any kind of extension to how the archive is being generated.

For now I can imagine four approaches (but more ideas are very welcome):

• Create a derivative of the current maven-jar-plugin implementation allowing for a timestamp=<number> attribute which is then used for all entries inserted into the jar file. If not set, the current behavior is kept.
• Revise the Jenkins fingerprinting scheme so it knows about jar files and only looks at the entries contents, not their metadata.
• Attach a plugin to the prepare-package stage responsible for touch ing the files with a specific time stamp. This requires all files to be present at that time (meaning that the jar plugin cannot be allowed to touch the MANIFEST.MF file)
• Attach an extra plugin to the "package" phase which rewrites the finished jar file, zeroing out all zip entry timestamps in the process.

Again, the goal is to make maven SNAPSHOT artifacts fully time independent so given the same source you get an artifact with the same MD5 checksum. I also believe, however, that this could be beneficial for release builds.

How should I approach this?

Answer 1

As per my comment, I still think the answer is to do none of the things you suggest, and instead use releases in preference to snapshots for artifacts which you are in fact releasing to customers.

The problems you describe are:

• you have a multi-module project which takes a long time to build because you have more than 100 modules,
• you have two snapshot artifacts which you think ought to be identical (because the source code and metadata were identical at build time), but they have different checksums.

My experience with Maven tells me that if you try and adhere to the "Maven Way", tools will work well for you out-of-the-box, but if you deviate then you'll have a bad time. Unfortunately, the Maven Way is sometimes elusive :-)

Multi-module projects in Maven are very useful when you have families of modules with code that varies in sympathy , eg you have a module containing a bunch of interfaces, and some sibling modules providing implementations. It would be unusual to have more than a dozen modules in a multi-module project. All the modules ought to share the version number of the parent (Maven doesn't enforce this, which in my opinion is confusing).

When you build a snapshot version of a multi-module project, snapshots of all modules are built, even if the code in a particular module hasn't changed. Therefore you can look at a family of modules in your repositiory, and know that at compile time the inter-module code references were satisfied.

For example, in a domain model module you might have an interface:

public interface Student {
    void study();
}

and in some sibling modules, which would declare compile-scoped dependencies on the domain model in their POMs, you might have implementations.

If you were then to change the interface in the domain model module:

public interface Student {
    void study();
    void drink(Beer beer);
}

and rebuild the multi-module project, the build will fail. The dependent modules will fail to build, even though their code and POMs have remained the same. In a multi-module project, you only install or deploy artifacts if all the child modules build successfully, so rebuilding snapshots is usually very desirable - it's telling you something about the inter-module dependencies.

If:

• you have an excessive number of modules, and/or
• those modules can't reasonably share the same version number, and/or
• you don't need any guarantees about code references between modules,

then your modularisation is incorrect. Don't use multi-module projects as a build system (you have Jenkins for that), use it instead to express relationships between modules of your code.

In your comment, you say:

RELEASE artifacts behave the same way when being rebuilt by Jenkins.

The point of point of release artifacts is that you do not rebuild them - they are definitive! If you use something like Artifactory, you will find that you cannot deploy a release artifact more than once - your Jenkins job should fail if you attempt it.

This is a fundamental tenet in Maven. One of the aims of Maven is that it if two developers on separate workstations were to attempt the same release, they would build artifacts which were functionally identical. If you are build an artifact which expresses a dependency (maybe for compilation purposes, or because it's being assembled into .war etc.) on another, then:

• if the dependency is a snapshot, Maven might seek a newer version from the repository.
• if the dependency is a release, the version in your local repository is assumed to be definitive.

If you could rebuild a release artifact, you would create the possibility that two developers have dissimilar versions in their repository, and you'd have dissimilar builds depending on which workstation you used. Don't do it.

Another critical detail is that a release artifact cannot depend on snapshot artifacts, again, you would lose various guarantees.

Releases are definitive, and it sounds like you want your assembly to depend on definitive artifacts. Jenkins makes tagging and releasing multi-module projects very straightforward.

In summary:

• Check your modularisation: one enormous multi-module project is not useful.
• If you don't want to continually rebuild snapshots, you need to do releases.
• Never release snapshots to your customer.
  • Follow the dependency graph of your assembly project and release any snapshots.
  • Release the assembly project, bumping your minor version.
  • Ensure your customer refers to the complete version number of your assembly in communications.