The difficult problem of managing Java dependencies

Dependency management is a difficult topic, much more difficult than most developers probably realize.

As long as things work, you barely need to pay any attention to which versions of your dependencies you’re currently using (but you should, of course), so that’s understandable.

But if you want to build reliable software while keeping up with latest security patches in all of your dependencies, which requires constantly updating libraries and making sure no breaking changes have been accidentally introduced and that all libraries remain compatible with each other, then you’re in for a challenge.

While this problem exists in every programming language in existence (with perhaps one exception), some languages do it better than others.

However, in this blog post, I am going to look in detail at the Java way of handling dependencies, which is mostly based on Maven (though Apache Ivy, Gradle and Bazel are also going to make appearances).

I will also mention some other language’s and language-agnostic package managers (like Nix, for example) in order to try to identify more clearly how things are done today outside the Java world, and where things seem to be heading.

Maven dependency management

Nearly everyone doing Java development is familiar with Maven. It’s by far the most popular build and dependency management tool for Java projects. However, I would dare say most developers don’t know how exactly Maven does dependency resolution (if you do, congratulations, you’re very likely in the minority, at least in my experience working with lots of other Java developers over more than a decade!).

Let’s look at a simple example, then try to guess what Maven will do with increasingly more complex cases.

To get started quickly with Maven, you just need to have it installed, then run this very simple command to generate a small project:

mvn archetype:generate -DgroupId=com.mycompany.app -DartifactId=my-app -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4 -DinteractiveMode=false

This will download about 100 files (between XML, POMs and jars), which if you’re not familiar with Maven, may sound excessive (because it is). But you’ll get a kind-of simple Java project that has the following structure:

└── my-app
    ├── pom.xml
    └── src
        ├── main
        │   └── java
        │       └── com
        │           └── mycompany
        │               └── app
        │                   └── App.java
        └── test
            └── java
                └── com
                    └── mycompany
                        └── app
                            └── AppTest.java

The important file for our purposes is the pom.xml file, which defines the project object model. The most relevant part of the POM is the dependencies it declares on external libraries.

The generated pom.xml looks like this:

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.mycompany.app</groupId>
  <artifactId>my-app</artifactId>
  <version>1.0-SNAPSHOT</version>

  <name>my-app</name>
  <!-- FIXME change it to the project's website -->
  <url>http://www.example.com</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.7</maven.compiler.source>
    <maven.compiler.target>1.7</maven.compiler.target>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.11</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <build>
    <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
      <plugins>
        <!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
        <plugin>
          <artifactId>maven-clean-plugin</artifactId>
          <version>3.1.0</version>
        </plugin>
        <!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
        <plugin>
          <artifactId>maven-resources-plugin</artifactId>
          <version>3.0.2</version>
        </plugin>
        <plugin>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>3.8.0</version>
        </plugin>
        <plugin>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>2.22.1</version>
        </plugin>
        <plugin>
          <artifactId>maven-jar-plugin</artifactId>
          <version>3.0.2</version>
        </plugin>
        <plugin>
          <artifactId>maven-install-plugin</artifactId>
          <version>2.5.2</version>
        </plugin>
        <plugin>
          <artifactId>maven-deploy-plugin</artifactId>
          <version>2.8.2</version>
        </plugin>
        <!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
        <plugin>
          <artifactId>maven-site-plugin</artifactId>
          <version>3.7.1</version>
        </plugin>
        <plugin>
          <artifactId>maven-project-info-reports-plugin</artifactId>
          <version>3.0.0</version>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
</project>

Yep, that’s 76 lines of XML for a basic project. It comes with a single dependency already added by default, on an awfully outdated JUnit 4.11 library.

To actually download that and build our project, we need run the famous Maven command:

mvn package

After somehow deciding to download another 121 files, Maven has packaged and tested our project. Interestingly, it did NOT download JUnit 4.11, probably because I already had that cached in my Maven Local repository (at ~/.m2/repo/).

We can confirm that Maven knows our project has a test dependency on that, though, by running the mvn dependency:tree command:

▶ mvn dependency:tree 
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-install-plugin/2.5.2/maven-install-plugin-2.5.2.pom

... ommited a few hundred lines here ...

Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-invoker/2.0.11/maven-invoker-2.0.11.jar (29 kB at 111 kB/s)
[INFO] com.mycompany.app:my-app:jar:1.0-SNAPSHOT
[INFO] \- junit:junit:jar:4.11:test
[INFO]    \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.990 s
[INFO] Finished at: 2022-01-29T18:49:51+01:00
[INFO] ------------------------------------------------------------------------

This required a hundred of so more downloads! But if you look carefully at the output, near the end, you’ll see it shows us the dependency tree of our project. Let’s separate that out so we can actually look carefully at it:

com.mycompany.app:my-app:jar:1.0-SNAPSHOT
\- junit:junit:jar:4.11:test
   \- org.hamcrest:hamcrest-core:jar:1.3:test

The dependency tree may actually be a graph, as some people will loudly interject as if that were some deep revelation that changes everything. That’s because each dependency of a project can appear in more than one “location”. However, graphs have no root, and dependencies always start at a root… so, is a dependency graph a graph with a root? Or a tree that allows nodes to be repeated? You decide!

As you can see, we actually have 2 test dependencies, junit and hamcrest-core.

If you go to Maven Central and look for the JUnit 4.11 POM, you’ll see that it declares the Hamcrest dependency, as shown by Maven:

<dependency>
   <groupId>org.hamcrest</groupId>
   <artifactId>hamcrest-core</artifactId>
   <version>1.3</version>
   <scope>compile</scope>
</dependency>

So far, so good.

To see more intricate cases, let’s create another project in a sibling directory and publish it to Maven Local:

cd ..
mvn archetype:generate -DgroupId=com.othercompany.app -DartifactId=other-app -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=1.4 -DinteractiveMode=false
cd other-app
mvn install
cd ../my-app

Now, we can add it as a dependency to this project:

<dependency>
    <groupId>com.othercompany.app</groupId>
    <artifactId>other-app</artifactId>
    <version>1.0-SNAPSHOT</version>
</dependency>

Running mvn dependency:tree again should show the change:

com.mycompany.app:my-app:jar:1.0-SNAPSHOT
+- com.othercompany.app:other-app:jar:1.0-SNAPSHOT:compile
\- junit:junit:jar:4.11:test
   \- org.hamcrest:hamcrest-core:jar:1.3:test

To see anything more interesting, we need quite a few dependencies. So, let’s make two more projects like this so that we have a dependency tree that looks like this:

Notice that I removed the SNAPSHOT from the versions. That’s because we are mostly concerned about stable dependencies in real-world projects and using snapshots would just complicate matters as Maven uses special rules for those.

com.mycompany.app:my-app:jar:1.0
+- com.othercompany.app:other-app:jar:1.0:compile
|  \- com.thirdcompany.app:third-app:jar:1.0:compile
|     \- com.fourthcompany.app:fourth-app:jar:1.0:compile
\- junit:junit:jar:4.11:test
   \- org.hamcrest:hamcrest-core:jar:1.3:test

I also published versions 1.1 and 2.0 of each project, with all projects matching each other’s versions for now (i.e. myapp:1.0 depends on another-app:1.0, my-app:1.1 depends on another-app:1.1 and so on.), so we can do some experiments:

1.1:

com.mycompany.app:my-app:jar:1.1
+- com.othercompany.app:other-app:jar:1.1:compile
|  \- com.thirdcompany.app:third-app:jar:1.1:compile
|     \- com.fourthcompany.app:fourth-app:jar:1.1:compile
\- junit:junit:jar:4.11:test
   \- org.hamcrest:hamcrest-core:jar:1.3:test

2.0:

com.mycompany.app:my-app:jar:2.0
+- com.othercompany.app:other-app:jar:2.0:compile
|  \- com.thirdcompany.app:third-app:jar:2.0:compile
|     \- com.fourthcompany.app:fourth-app:jar:2.0:compile
\- junit:junit:jar:4.11:test
   \- org.hamcrest:hamcrest-core:jar:1.3:test

To intentionally create a conflict, let’s go back to version 1.1 of my-app, but add a direct dependency on fourth-app version 1.0 to it. It should then conflict with third-app’s dependency on fourth-app with version 1.1.

What does Maven do in this case?

Let’s check it:

com.mycompany.app:my-app:jar:1.1
+- com.othercompany.app:other-app:jar:1.1:compile
|  \- com.thirdcompany.app:third-app:jar:1.1:compile
+- com.fourthcompany.app:fourth-app:jar:1.0:compile
\- junit:junit:jar:4.11:test
   \- org.hamcrest:hamcrest-core:jar:1.3:test

Notice how third-app seems to have lost its dependency on fourth-app, and the direct dependency from my-app to fourth-app was honoured with the version it requested!

This could, of course, break third-app, as it was likely not tested with an older version of fourth-app… even with semantic versioning being followed religiously, this case could cause a breakage because fourth-app could’ve added new functionality in version 1.1 (which is not a breaking change) on which third-app 1.1 depended on to work!

The reason Maven does this is that it always tries to honour the exact version of a conflicting dependency that is closest to the root of the dependency tree. In this case, because fourth-app is a direct dependency of my-app, whatever version is declared for fourth-app in my-app is honoured, regardless of any other transitive dependency on it.

We can go even further and say that my-app now depends on fourth-app version 2.0, which is likely to break third-app version 1.1 even more (as it’s free to remove, change or even completely change package names between major versions)!

com.mycompany.app:my-app:jar:1.1
+- com.othercompany.app:other-app:jar:1.1:compile
|  \- com.thirdcompany.app:third-app:jar:1.1:compile
+- com.fourthcompany.app:fourth-app:jar:2.0:compile
\- junit:junit:jar:4.11:test
   \- org.hamcrest:hamcrest-core:jar:1.3:test

Maven doesn’t care, it does the same thing again and it won’t even issue a warning.

What does Gradle do in this case, you may ask.

Let’s try it:

build.gradle:

plugins {
    id 'java-library'
}

group 'com.mycompany.app'
version '1.1'

repositories {
    mavenLocal()
}

dependencies {
    implementation 'com.othercompany.app:other-app:1.1'
    implementation 'com.fourthcompany.app:fourth-app:2.0'
}

Running it:

▶ gradle dependencies --configuration=runtimeClasspath
Starting a Gradle Daemon (subsequent builds will be faster)

> Task :dependencies

    ------------------------------------------------------------
Root project 'my-app'
------------------------------------------------------------

runtimeClasspath - Runtime classpath of source set 'main'.
+--- com.othercompany.app:other-app:1.1
|    \--- com.thirdcompany.app:third-app:1.1
|         \--- com.fourthcompany.app:fourth-app:1.1 -> 2.0
\--- com.fourthcompany.app:fourth-app:2.0

(*) - dependencies omitted (listed previously)

A web-based, searchable dependency report is available by adding the --scan option.

BUILD SUCCESSFUL in 5s
1 actionable task: 1 executed

Gradle may not download hundreds of files to run commands, but it has to start a daemon in the background to look faster in the subsequent runs… even Maven has its own daemon available now (at least it’s not the default) and the Kotlin compiler too… Java tools are becoming really annoying like that, pretending I have all the RAM in the world (I don’t, I use a little Macbook Air with 8GB of RAM and that’s far from enough these days because of all these deamons for everything).

Gradle does the exact same thing as Maven by default, but it at least shows that it is auto-magically allowing a pretty big version change in the third-app’s dependency: fourth-app:1.1 -> 2.0.

For the record, grandpa Ivy also does the same thing. Adding an ivy.xml file that looks like this (besidses configuring a settings file so that Ivy looks at the Maven local repository):

<ivy-module version="2.0">
    <info organisation="com.mycompany" module="my-app"/>
    <dependencies>
      <dependency org="com.othercompany.app" name="other-app" rev="1.1"/>
      <dependency org="com.fourthcompany.app" name="fourth-app" rev="2.0"/>
    </dependencies>
</ivy-module>

Running it (see how it says 1 dependency was evicted?):

▶ java -jar ivy/ivy-2.5.0.jar -settings ivysettings.xml resolve
:: loading settings :: file = ivysettings.xml
:: resolving dependencies :: com.mycompany#my-app;working@Renatos-Air
	confs: [default]
	found com.othercompany.app#other-app;1.1 in maven2
	found com.thirdcompany.app#third-app;1.1 in maven2
	found junit#junit;4.11 in maven2
	found org.hamcrest#hamcrest-core;1.3 in maven2
	found com.fourthcompany.app#fourth-app;2.0 in maven2
:: resolution report :: resolve 148ms :: artifacts dl 12ms
	:: evicted modules:
	com.fourthcompany.app#fourth-app;1.1 by [com.fourthcompany.app#fourth-app;2.0] in [default]
	---------------------------------------------------------------------
	|                  |            modules            ||   artifacts   |
	|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
	---------------------------------------------------------------------
	|      default     |   6   |   0   |   0   |   1   ||   5   |   0   |
	---------------------------------------------------------------------

But going back to Gradle…

If we want Gradle to actually care about not trying to break our dependencies like that, silently, we need to configure a ResolutionStrategy.

Adding this to build.gradle:

configurations.all {
  resolutionStrategy {
    failOnVersionConflict()
  }
}

Causes the build to fail now:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':dependencies'.
> Could not resolve all dependencies for configuration ':runtimeClasspath'.
   > Conflict(s) found for the following module(s):
       - com.fourthcompany.app:fourth-app between versions 2.0 and 1.1
     Run with:
         --scan or
         :dependencyInsight --configuration runtimeClasspath --dependency com.fourthcompany.app:fourth-app
     to get more insight on how to solve the conflict.

Maven can also detect this problem if the Maven Enforcer Plugin is used.

Typical Maven usability issue, however: if you just run mvn dependency:tree, it will happily report success! You need to actually build the project to see the problem:

▶ mvn package
[INFO] Scanning for projects...
[INFO] 
[INFO] ----------------------< com.mycompany.app:my-app >----------------------
[INFO] Building my-app 1.1
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-enforcer-plugin:3.0.0:enforce (enforce) @ my-app ---
[WARNING] 
Dependency convergence error for com.fourthcompany.app:fourth-app:jar:1.1:compile paths to dependency are:
+-com.mycompany.app:my-app:jar:1.1
  +-com.othercompany.app:other-app:jar:1.1:compile
    +-com.thirdcompany.app:third-app:jar:1.1:compile
      +-com.fourthcompany.app:fourth-app:jar:1.1:compile
and
+-com.mycompany.app:my-app:jar:1.1
  +-com.fourthcompany.app:fourth-app:jar:2.0:compile

[WARNING] Rule 0: org.apache.maven.plugins.enforcer.DependencyConvergence failed with message:
Failed while enforcing releasability. See above detailed error message.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.665 s
[INFO] Finished at: 2022-01-29T20:35:49+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce (enforce) on project my-app: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

▶ mvn dependency:tree
[INFO] Scanning for projects...
[INFO] 
[INFO] ----------------------< com.mycompany.app:my-app >----------------------
[INFO] Building my-app 1.1
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ my-app ---
[INFO] com.mycompany.app:my-app:jar:1.1
[INFO] +- com.othercompany.app:other-app:jar:1.1:compile
[INFO] |  \- com.thirdcompany.app:third-app:jar:1.1:compile
[INFO] +- com.fourthcompany.app:fourth-app:jar:2.0:compile
[INFO] \- junit:junit:jar:4.11:test
[INFO]    \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.030 s
[INFO] Finished at: 2022-01-29T20:35:55+01:00
[INFO] ------------------------------------------------------------------------

With Maven/Gradle/Ivy, it’s still possible to fix the issue, as we’ll see in the next section, but you’ll need to manually apply version overrides, which is a tedious, error-prone and fragile approach because every time you change your library versions (which should happen often) you’ll need to make sure to check each version override again, manually (most likely it’ll just remain there even if it becomes stale or unnecessary… until you run into trouble again, of course).

Bazel has a different approach to the problem: it forces developers to declare ALL dependencies of their project and refuses to resolve any transitive dependency.

As the Bazel docs say:

Bazel only reads dependencies listed in your WORKSPACE file.
If your project (A) depends on another project (B) which lists a dependency on a
third project (C) in its WORKSPACE file, you’ll have to add both B and C to your
project’s WORKSPACE file.
This requirement can balloon the WORKSPACE file size, but limits the chances of
having one library include C at version 1.0 and another include C at 2.0.

I don’t know how I feel about that. On the one hand, this forces developers to confront the fact that transitive dependencies exist and can easily clash, but on the other hand it’s like throwing the baby out with the bathwater and just giving up on automation of dependency resolution entirely.

Maybe there is a better way?!

Maven and Gradle have ways to solve conflicts manually. This Baeldung article explains how to auto-upgrade dependencies with Maven. See also this StackOverflow question. Gradle offers sophisticated ways to resolve conflicts. It’s a rabbit hole and by the time you get out, you might not even remember what you were trying to do, however.

The “modern” approach.

Most newer languages’ package managers tend to use dependency ranges instead of fixed dependencies versions like Maven/Gradle projects usually do (as we’ll see, both can actually also use dependency ranges, they’re just not as common).

In npm, your dependencies nearly always would look like this:

{
  "name": "my-app",
  "version": "1.0",
  "dependencies": {
    "base64-js": "^1.5.1",
    "random-words": "^1.1.2"
  }
}

Where ^1.5.1 means the latest minor version from 1.5.1 onwards. That means every time npm install is run, it will look at the registry and see what’s the latest minor version after the specified version, and use that.

That would be obviously unreliable if it stayed at just that, but npm uses a lock file to guarantee that once the versions are resolved, they remain frozen in another file which is then used in subsequent builds.

That may sound pointless, but allows developers to easily upgrade dependencies automatically by removing the lock file and running npm install again (or just running whatever command does that automatically for them).

In a world where dependencies change at a fast pace and where security vulnerabilities happen every other day, this kind of mechanism becomes almost a necessity, despite the obvious difficulty that causes in keeping software working properly unless everyone, everywhere, follows semantic versioning to the letter and make sure that their tests are comprehensive enough to allow a dynamically typed language to update dependencies and still continue working as if nothing happened. As we know, in reality, things actually break all the time, to no one’s surprise.

Another modern language whose package manager works similarly to npm is Rust with its Cargo package manager.

In npm, conflicts as we saw in the previous section are not even attempted to be fixed (both versions end up in node_modules/, and each package sees its own version of the library - and pray for actual exchanged functions and objects to still remain compatible between them).

With Cargo, it works similarly!

From the Cargo Book:

If multiple packages have a common dependency with semver-incompatible versions,
then Cargo will allow this, but will build two separate copies of the dependency.

Notice that in Cargo, specifying a dependency like rand = "0.7" is exactly equivalent to declaring it as rand = "^0.7". To declare a fixed version, it must be specified as rand = "=0.7" (which I believe is almost never done and probably even frowned upon).

The Cargo behaviour is remarkably similar to how OSGi works in the Java world: it allows multiple versions of a dependency to co-exist in the JVM runtime as long as they do not directly interact… but to my knowledge, this is not ensured by Cargo until a conflict actually happens at runtime (at which time your application dies an unhappy death, known as panic in Rust), but OSGi will throw an error as it wires up the bundles (typically at startup) if it detects two packages interacting with different versions of the same bundle.

While Rust is usually a model for what’s the right thing to do, I have to say that in this aspect, it’s not really doing anything revolutionary, just being pragmatic about it. Unfortunately, its approach is not applicable in the JVM without classpath isolation (like OSGi) or JMS module layers isolation, as a new project called Layrry is trying to do.

A note on committing lock files to source control:

Package managers that use lock files seem to have converged on the idea that libraries should not commit their lock files, only applications should do that. As the Rust Book explains, only end products like binaries have a full picture to decide what versions of dependencies should be used.

The Dart Dev Guide also recommends committing the lock file only for application packages.

I would recommend looking at Pub, the Dart/Flutter Package Manager, for a good example of how to make dependency management easier and more visible, specially with the pub outdated and pub upgrade commands, the scoring system to help judge the quality of a library, and the fact that one can declare in the pubspec the versions of the Dart runtime the library is meant to run with, and which of the platforms (OS, web, Flutter) the library explicitly wants to support (as opposed to which platforms the source code supports, which is automatically detected).

This may pose a problem for Continous Integration (CI), as it will cause every CI build to potentially resolve different versions for the dependencies of a project. However, because when you use dependency version ranges, you’re basically promising to consumers of your library that your library will work with any set of packages whose versions meet the ranges described in your package, this is actually not supposed to be a problem. Right? Well, if you care about reproducible builds, you might disagree with that.

Check Useless lying version ranges for a stronger case against version ranges.

Using version ranges in Maven/Gradle/Ivy

I have very rarely seen Java projects using version ranges like other ecosystems prefer to do. But as luck would have it (and this is what motivated me to write this article), the other day, while I was trying to fix a security vulnerability at work, I noticed that the dependency tree of one of our external dependencies specified all of its versions as version ranges.

The Maven - Introduction to the Dependency Mechanism article does not even mention that Maven supports version ranges.

You can see in the build.gradle file for the java-webauthn-server project looks like this:

dependencies {
  constraints {
    api('ch.qos.logback:logback-classic:[1.2.3,2)')
    api('com.augustcellars.cose:cose-java:[1.0.0,2)')
    api('com.fasterxml.jackson.core:jackson-databind:[2.11.0,3)')
    ...
  }
}

The POM on Maven Central actually has the versions declared exactly as shown above. This is not Gradle-specific.

However, as Maven does not employ a lock file (though it seems it’s possible to do so) this seems exceedingly dangerous. It’s basically impossible to tell whether the library versions you test with are actually what will be used when someone builds a project depending on this library.

Gradle has support for lock files, but it only works when version ranges are used. And as we’ve seen, Gradle has unexpected behaviour when version ranges are used anyway. I am still waiting to see a gradle.lockfile in the wild. Together with publishing resolved versions (so it’s not necessary to understand lock files when resolving dependencies) this is actually an interesting feature, though easily replaceable with the much simpler Versions Plugin, which just updates versions directly in the build file so you actually have only one place to look for that.

I understand the good intention here: as this library is in the field of security, it attempts to make sure all of its dependencies are always up-to-date. But all it takes is a small slip by the developers of the many dependencies for everything to fall down on the next unintended, automatic minor update.

In fact, the vulnerability I was looking at (which I prefer to not mention here as that’s not really relevant for this post) was in a pretty old transitive dependency of this library which was NOT resolving to the latest minor release (as intended by the authors of this build.gradle file) because another library in the dependency tree of this project had a fixed version declared for that. In such cases, Gradle always seems to use the fixed version.

You can read the detailed logic of how Gradle handes cases like this here. It claims to always use the highest version by default in case there are no version ranges, but prefers to use a non-range version if available, as is the case in this example.

We can prove that by changing our my-app’s build.gradle file again to depend on fourth-app with a version range [1.0,2.0] (inclusive on both ends!), while reverting the other dependencies to version 1.0, which should result in the full tree using version 1.0, except for fourth-app, which is allowed to use 2.0:

dependencies {
    implementation 'com.othercompany.app:other-app:1.0'
    implementation 'com.fourthcompany.app:fourth-app:[1.0,2.0]'
}

Gradle, surprisingly to me, uses fourth-app:1.0 (with or without using failOnversionconflict):

+--- com.othercompany.app:other-app:1.0
|    \--- com.thirdcompany.app:third-app:1.0
|         \--- com.fourthcompany.app:fourth-app:1.0
\--- com.fourthcompany.app:fourth-app:[1.0,2.0] -> 1.0

While Maven resolves it as one would expect, as allowed by the version range:

com.mycompany.app:my-app:jar:1.0
+- com.othercompany.app:other-app:jar:1.0:compile
|  \- com.thirdcompany.app:third-app:jar:1.0:compile
+- com.fourthcompany.app:fourth-app:jar:2.0:compile

To break the tie… Ivy does the same thing as Maven:

:: resolving dependencies :: com.mycompany#my-app;working@Renatos-Air
	confs: [default]
	found com.othercompany.app#other-app;1.0 in maven2
	found com.thirdcompany.app#third-app;1.0 in maven2
	found com.fourthcompany.app#fourth-app;1.0 in maven2
	found junit#junit;4.11 in maven2
	found org.hamcrest#hamcrest-core;1.3 in maven2
	found com.fourthcompany.app#fourth-app;2.0 in maven2
	[2.0] com.fourthcompany.app#fourth-app;[1.0,2.0]
:: resolution report :: resolve 213ms :: artifacts dl 9ms
	:: evicted modules:
	com.fourthcompany.app#fourth-app;1.0 by [com.fourthcompany.app#fourth-app;2.0] in [default]

Not that doing this is much better: it is just blindly throwing packages together and hoping that by some miracle things will actually work despite the versions not having been tested together. There has to be a better way.

Nix

The NixOS project, with its Nix Package Manager, has been touted as the package manager of the future. It describes itself as a purely functional package manager.

From the Nix documentation section How it works: “This means that it treats packages like values in purely functional programming languages such as Haskell — they are built by functions that don’t have side-effects, and they never change after they have been built.”

Unfortunately, to the best of my knowledge, this means Nix packages are treated as completely self-contained entities, much like static binaries. That also means it solves the problem of transitive dependencies by completely removing their existence!

This may sound great for binary packages you install on your Operating System, but in the context of building programming language packages and applications, that doesn’t really work well. It would be akin to Java jars shading every dependency they use. And in case they shared some of their dependencies’ APIs in their own APIs, you would run into the problem of their types, despite coming from the same library, not being compatible even if the same version had been used in different packages, so you would need to have support for unshading shared libraries, which would make transitive dependencies a thing again, and you’re back to square one.

Java Module System

In Java 9, the Java Module System was introduced in order to make a set of packages typically compiled together and put into a jar (which is how Java code has been packaged since Java 1) a recognizable entity in the Java Language Specification, and something tangible at runtime. It basically adds an additional scope encapsulation level to the Java language besides methods, classes and packages.

It is part of the design of Java modules that modules should be identified by their name alone, which means that their versions are not part of their identity. In practice, this means that only one version of a module can be resolved at runtime (to be more precise, it’s one module per module layer, so it is actually possible to have more than one version of a module loaded in different layers - something that few Java projects seem to consider doing as of 2022, but which, as mentioned earlier, may be made easier by projects like Layrry in the future).

I am mentioning Java modules here for completeness as it may appear that they could help with dependency management problems, but notice that they do not address in any way how dependency management works other than preventing a Java modular application from starting up in case any required module is absent (which is a great improvement in itself). Whether the modules have the right versions to work together, or even how dependent modules are found in the first place, are completely left out of the Java modules system specification and left to systems like Maven to solve.

What else can be done?

So, should we give up and accept things are too difficult and there’s nothing we can do about it?

Well, I believe that, at least for Java, there is a way out of this mess.

I have been working on a new tool called JBuild which has the objective of resolving dependencies conflicts in a novel way.

JBuild is still in early development. What I am showing here already works, but there are many things I want to improve on before releasing it to the general public. If you want to try it out anyway and believe in the vision I put forward here, visit the link above and create issues or send pull requests!

Instead of looking at version numbers, JBuild actually looks at the bytecode of each jar after resolving the dependency tree without removing any conflicts. Given one or more entry-points (the jars that contain the main for an application), it will traverse the AST of each class in those entry-points, looking for references it requires and trying to find them in the other jars present in the candidate classpaths, recursively, until it can prove that a set of jars can work together well, at least at the type-level (checking semantics are correct would be an intractable problem, unfortunately).

JBuild first needs to compute all possible permutations of jars that are possible to put in the same dependency tree.

For example, resolving the dependency tree for my-app version 1.0 that has a direct dependency on fourth-app:2.0 results in this tree:

▶ jbuild deps -t com.mycompany.app:my-app:1.0
Dependencies of com.mycompany.app:my-app:1.0 (incl. transitive):
  - scope compile
    * com.fourthcompany.app:fourth-app:2.0 [compile]
    * com.othercompany.app:other-app:1.0 [compile]
        * com.thirdcompany.app:third-app:1.0 [compile]
            * com.fourthcompany.app:fourth-app:1.0 [compile]
  The artifact com.fourthcompany.app:fourth-app is required with more than one version:
    * 2.0 (com.fourthcompany.app:fourth-app:2.0)
    * 1.0 (com.othercompany.app:other-app:1.0 -> com.thirdcompany.app:third-app:1.0 -> com.fourthcompany.app:fourth-app:1.0)
  4 compile dependencies listed
  - scope test
    * junit:junit:4.11 [test]
        * org.hamcrest:hamcrest-core:1.3 [compile]
  2 test dependencies listed
JBuild success in 294 ms!

It detects and clearly shows the conflict.

However, when the install command is used (which downloads the transitive dependency tree with runtime scope for one or more artifacts), JBuild doesn’t bother to resolve any conflicts and just installs both versions in the output directory:

▶ jbuild install com.mycompany.app:my-app:1.0 -d my-app
Will install 5 artifacts at my-app
Successfully installed 5 artifacts at my-app
JBuild success in 295 ms!

▶ ls my-app
fourth-app-1.0.jar
fourth-app-2.0.jar
my-app-1.0.jar
other-app-1.0.jar
third-app-1.0.jar

This would be bad if it just ended there… but JBuild has another command, tentatively called doctor, that can then look at the bytecode and actually find a working set of jars, given our application’s main is on my-app (which means there may be missing types and methods in other jars, which is very common in the Java world with optional dependencies, but all requirements of the entry-point jars must be fulfilled):

▶ jbuild doctor my-app -y -e my-app-1.0.jar
The following jars conflict:
  * fourth-app-2.0.jar, fourth-app-1.0.jar
Detected conflicts in classpath, resulting in 2 possible classpath permutations. Trying to find a consistent permutation.
All entrypoint type dependencies are satisfied by the classpath below:

my-app/my-app-1.0.jar

JBuild success in 271 ms!

Notice that the doctor command, as the deps command, detects the conflict between the two versions of the “same” library… but the doctor command knows that they conflict not because it knows the libraries have the same coordinates (it can’t even see the POM file, as its input is a directory with jars, and it does not try to guess the name of the package from the jar name) but because it can see the two jars have intersecting types, a clear sign of two jars representing the same library.

It correctly detects that the Maven dependencies were actually all unnecessary because our jar has no bytecode dependencies on them! So the only jar required to run any code in this application is the application jar itself.

To see something more realistic, we need to actually simulate the scenarios that were being hypothesized so far.

Let’s define the simplest possible types for the 4 libraries being used so far, starting with all libraries on version 1.0:

// fourth-app 1.0
package com.fourthcompany.app;

public class Fourth {
    public String getMessage() { return "hello"; }
}

// third-app 1.0
package com.thirdcompany.app;
import com.fourthcompany.app.Fourth;

public class Third {
    public String upperCaseMessage() {
        return new Fourth().getMessage().toUpperCase();
    }
}

// second-app 1.0
package com.othercompany.app;
import com.thirdcompany.app.Third;

public class Second {
    public String wrapMessageIn(String delimiter) {
        return delimiter +
            new Third().upperCaseMessage() +
            delimiter;
    }
}

// my-app 1.0
package com.mycompany.app;
import com.secondcompany.app.Second;

public class App {
    public static void main(String... args) {
        var message = new Second().wrapMessageIn("|");
        System.out.println(message);
    }
}

Now, let’s remember the first possible conflicting scenario mentioned earlier:

To intentionally create a conflict, let's go back to version 1.1 of `my-app`, but add a direct dependency
on `fourth-app` version `1.0` to it.
It should then conflict with `third-app`'s dependency on `fourth-app` with version `1.1`.

Alright, so let’s make MyApp use Fourth version 1.0 directly, while at the same time upgrading Third to version 1.1, failing to realise Third 1.1 actually changed so that it can’t work with Fourth 1.0 anymore, only with 1.1:

Notice that Third is not a direct dependency of the main project. It’s required by other-app, so this scenario could easily happen in real life.

// fourth-app 1.1
package com.fourthcompany.app;

public class Fourth {
    public String getMessage() { return "hello"; }
    // new in 1.1 (semver OK)
    public String getMessage(boolean arriving) {
        return arriving ? "hello" : "bye";
    }
}

// third-app 1.1
package com.thirdcompany.app;
import com.fourthcompany.app.Fourth;

public class Third {
    public String upperCaseMessage() {
        // the dev decided to use the new method from v1.1!
        return new Fourth().getMessage(true).toUpperCase();
    }
}

// my-app 1.1
package com.mycompany.app;
import com.secondcompany.app.Second;

public class App {
    public static void main(String... args) {
        var message = new Second().wrapMessageIn("|");
        // dev decided to use fourth v1.0 directly, but
        // upgraded to third 1.1, failing to notice it requires fourth 1.1
        message += new Fourth().getMessage();
        System.out.println(message);
    }
}

Commit with this change.

Maven/Gradle/Ant don’t care about the actual code, so nothing we’ve seen so far would change. But JBuild cares!

First, let’s check if the dependency tree shows the conflict:

▶ jbuild deps -t com.mycompany.app:my-app:1.1
Dependencies of com.mycompany.app:my-app:1.1 (incl. transitive):
  - scope compile
    * com.fourthcompany.app:fourth-app:1.0 [compile]
    * com.othercompany.app:other-app:1.1 [compile]
        * com.thirdcompany.app:third-app:1.1 [compile]
            * com.fourthcompany.app:fourth-app:1.1 [compile]
  The artifact com.fourthcompany.app:fourth-app is required with more than one version:
    * 1.1 (com.othercompany.app:other-app:1.1 -> com.thirdcompany.app:third-app:1.1 -> com.fourthcompany.app:fourth-app:1.1)
    * 1.0 (com.fourthcompany.app:fourth-app:1.0)
  4 compile dependencies listed
  - scope test
    * junit:junit:4.11 [test]
        * org.hamcrest:hamcrest-core:1.3 [compile]
  2 test dependencies listed
JBuild success in 289 ms!

Perfect! It does.

Let’s see what jbuild doctor says about this:

▶ jbuild install com.mycompany.app:my-app:1.1
Will install 5 artifacts at java-libs
Successfully installed 5 artifacts at java-libs
JBuild success in 300 ms!

▶ jbuild doctor java-libs -e my-app-1.1.jar
The following jars conflict:
  * fourth-app-1.1.jar, fourth-app-1.0.jar
All entrypoint type dependencies are satisfied by the classpath below:

java-libs/my-app-1.1.jar:java-libs/other-app-1.1.jar:java-libs/third-app-1.1.jar:java-libs/fourth-app-1.1.jar
JBuild success in 359 ms!

jbuild doctor can see that only one of the conflicting jars can actually be used assuming the my-app.jar is the entry-point (i.e. the jar containing the main application code). The other possible classpath, using fourth-app-1.0.jar, is discarded as its bytecode-incompatible… if it weren’t, JBuild would show all valid classpaths and the user would need to choose one of them.

Of course, in the real world, this wouldn’t be so easy. Let’s try to run the doctor command on a fairly complex application, the Spring Boot CLI:

▶ jbuild install org.springframework.boot:spring-boot-cli:2.6.1 -d spring 
Will install 55 artifacts at spring
Successfully installed 55 artifacts at spring
JBuild success in 1 sec, 604 ms!

▶ jbuild doctor spring -e spring/spring-boot-cli-2.6.1.jar 
The following jars conflict:
  * maven-resolver-spi-1.6.1.jar, maven-resolver-spi-1.4.1.jar
  * httpclient-4.5.13.jar, httpclient-4.5.12.jar
  * maven-resolver-api-1.6.1.jar, maven-resolver-api-1.4.1.jar
  * plexus-utils-3.2.1.jar, plexus-utils-1.5.5.jar
  * maven-resolver-util-1.6.1.jar, maven-resolver-util-1.4.1.jar
  * slf4j-api-1.7.32.jar, slf4j-api-1.7.30.jar
  * org.eclipse.sisu.inject-0.3.4.jar, org.eclipse.sisu.inject-0.3.0.jar
...

It’s quite amazing this CLI even runs. What are the chances that a library depending on plexus-utils-3.2.1 will work well with another that depends on plexus-utils-1.5.5?

If any Spring Boot maintainers are reading: you can run jbuild deps -t org.springframework.boot:spring-boot-cli:2.6.1 to print a full report of conflicts you’ve got in your classpath, which includes this:

  The artifact org.codehaus.plexus:plexus-utils is required with more than one version:
    * 3.0.18 (org.sonatype.sisu:sisu-inject-plexus:2.6.0 -> org.codehaus.plexus:plexus-utils:3.0.18)
    * 3.0.17 (org.sonatype.sisu:sisu-inject-plexus:2.6.0 -> org.eclipse.sisu:org.eclipse.sisu.plexus:0.3.0 -> org.codehaus.plexus:plexus-utils:3.0.17)
    * 3.2.1 (org.apache.maven:maven-model:3.6.3 -> org.codehaus.plexus:plexus-utils:3.2.1)
    * 1.5.5 (org.apache.maven:maven-settings-builder:3.6.3 -> org.sonatype.plexus:plexus-sec-dispatcher:1.4 -> org.codehaus.plexus:plexus-utils:1.5.5)

Kind of entertaining that the Plexus Security Dispatcher is so incredibly outdated… by upgrading maven-settings-builder to the more recent 3.8.4 version, you would get plexus-utils version 3.4.1 in that branch, which was released 12 years after version 1.5.5.

This shows how even in a professional project that has the attention of lots of people, dependencies can get really messy.

Conclusion

Java dependency management tools have an important job, but I feel like, somehow, they’re stuck in a local maximum. As this blog post shows, they do things sometimes that are not very far from guessing. Even if developers mostly followed semantic versioning (which to my surprise, from experience, is not everyone at all), as I’ve shown, these tools will still do unsound things by default, and tweaking them to do the right thing is non-trivial and requires a lot of specialized knowledge.

I have been working on JBuild on and off in the last 6 months or so. I believe the current state of dependency management in the Java world is far from optimal, and JBuild may have something to offer in the way of ideas to make the problem more managable.

I am not sure yet, to be honest, that this is the way to go, but as I hope I’ve shown above, JBuild is looking very promising.

If you have ideas or like what I’ve shown, do get in touch by opening an issue on the JBuild repository. I am always eager to hear what others have to say!