Fails to load dependency graph when dependency uses Version Range

[chrsmutti]

First of all, thanks for the amazing work on this tool, really easy to use and it'll be an incredible selling point for me to pitch Mill and a more modern build process for some services at work!

I was testing this in an internal repository and had some issues with the dependency graph not loading, after some debugging (adding projects incrementally and reading the output of generate), I've pinpointed the issue to Version Ranges.

One of the dependencies of this project is org.jongo:jongo:1.5.0, with depends on com.fasterxml.jackson.core:jackson-core:[2.7.0,2.12.3]. When the manifest is generated with this type of version range, the Dependency Graph page fails to load, downgrading to org.jongo:jongo:1.4.1 that does not include version ranges for Jackson works perfectly.

Github has some way of dealing with this, if the project has a pom.xml it shows the version range correctly, as seen here:

But I did not find anything on the documentation of the Dependency Submission API.

[ckipp01]

Hey @chrsmutti, thanks for the report! Yes! I hit on the in scala-cli as well and it was super confusing. It turns out that the submission API doesn't support ranges since you have to have a valid PURL, which don't support ranges. That's totally fine because in reality, we don't want to display a range here, we actually want to display the resolved result. I'm not sure what version of mill-github-dependency-graph you're using, but this should be fixed in 0.1.0 that I released yesterday. If you're not on the latest version, could you check with that? I also plan on actually adding in https://github.com/package-url/packageurl-java to help with this, because there are some other oddities in PURLs.

Also, if you haven't seen I also created https://github.com/ckipp01/mill-dependency-submission which should be bringing in the latest version with this fix.

[chrsmutti]

I'm actually using the dependency submission action already:

An excerpt of the generated manifest using ./millw --import ivy:io.chris-kipp::mill-github-dependency-graph::0.1.0 show io.kipp.mill.github.dependency.graph.Graph/generate:

      "com.fasterxml.jackson.core:jackson-databind:[2.7.0,2.12.3]": {
        "metadata": {
          
        },
        "dependencies": [
          "com.fasterxml.jackson.core:jackson-annotations:2.12.3",
          "com.fasterxml.jackson.core:jackson-core:2.12.3"
        ],
        "relationship": "indirect",
        "package_url": "pkg:maven/com.fasterxml.jackson.core/jackson-databind@[2.7.0,2.12.3]"
      }

[ckipp01]

huh, that's very odd. I just tested this with:

import mill._, scalalib._

object minimal extends ScalaModule {
  def scalaVersion = "2.13.8"

  def ivyDeps = Agg(
    ivy"org.jongo:jongo:1.5.0"
  )
}

And then when running generate on it I see that jackson-databind is indeed resolved:

      "com.fasterxml.jackson.core:jackson-databind:2.12.3": {
        "metadata": {

        },
        "dependencies": [
          "com.fasterxml.jackson.core:jackson-annotations:2.12.3",
          "com.fasterxml.jackson.core:jackson-core:2.12.3"
        ],
        "relationship": "indirect",
        "package_url": "pkg:maven/com.fasterxml.jackson.core/jackson-databind@2.12.3"
      },

Is this project public by chance that I could take a look at?

[chrsmutti]

Yeah, very odd indeed, I just tested a minimal version as well and had the same result as you. The project is internal 😞

But I think I've reproduced it, it may be related to a module dep adding in some other version of the dependency, like so:

import mill._, scalalib._

object minimalDep extends ScalaModule {
  def scalaVersion = "2.13.8"

  def ivyDeps = Agg(
    ivy"com.fasterxml.jackson.core:jackson-core:2.13.3",
  )
}

object minimal extends ScalaModule {
  def moduleDeps = Seq(`minimalDep`)

  def scalaVersion = "2.13.8"

  def ivyDeps = Agg(
    ivy"org.jongo:jongo:1.5.0"
  )
}

This architecture one is closer to the original repo I was testing it, and results in:

      "com.fasterxml.jackson.core:jackson-core:[2.7.0,2.12.3]": {
        "metadata": {

        },
        "dependencies": [

        ],
        "relationship": "indirect",
        "package_url": "pkg:maven/com.fasterxml.jackson.core/jackson-core@[2.7.0,2.12.3]"
      },

If moduleDep depends on com.fasterxml.jackson.core:jackson-core in the same range as org.jongo:jongo, the issue is also resolved. So it seems like an extreme edge case 😅

[lefou]

Question is, which version actually ends up in the classpath?

[ckipp01]

Ah perfect, thanks for the reproduction @chrsmutti. This is super helpful. I'll dig in and see what's going on because yea, this shouldn't be ending up here and I actually have no idea how it is since I'm grabbing the reconciled version every time. I'll dig in and report back.

[ckipp01]

Alright, so interesting enough, I think this might be a bug in Coursier's DependencyTrees. If you use resolve to test this out, you can see that it's resolved correctly with 2.12.3:

❯ cs resolve org.jongo:jongo:1.5.0
com.fasterxml.jackson.core:jackson-annotations:2.12.3:default
com.fasterxml.jackson.core:jackson-core:2.12.3:default
com.fasterxml.jackson.core:jackson-databind:2.12.3:default
de.undercouch:bson4jackson:2.12.0:default
org.jongo:jongo:1.5.0:default

However, with the minimal example you have, even if you try a mill minimal.ivyDepsTree you can see the issue:

❯ ./mill minimal.ivyDepsTree
[16/16] minimal.ivyDepsTree
├─ org.jongo:jongo:1.5.0
│  ├─ com.fasterxml.jackson.core:jackson-annotations:[2.7.0,2.12.3] -> 2.12.3
│  ├─ com.fasterxml.jackson.core:jackson-core:[2.7.0,2.12.3]
│  ├─ com.fasterxml.jackson.core:jackson-databind:[2.7.0,2.12.3] -> 2.12.3
│  │  ├─ com.fasterxml.jackson.core:jackson-annotations:2.12.3
│  │  └─ com.fasterxml.jackson.core:jackson-core:2.12.3
│  └─ de.undercouch:bson4jackson:2.12.0
├─ org.scala-lang:scala-library:2.13.8
└─ com.fasterxml.jackson.core:jackson-core:2.13.3

I don't believe I've ever seen a reconciled version in the tree still have the range like it does here. I'd expect core to also have the same -> 2.12.3, but it doesn't. The funny part is that when you dig into the actual tree, when I come across this dep I see:

dep version: [2.7.0,2.12.3]
retained version: [2.7.0,2.12.3]
reconciled version: [2.7.0,2.12.3]

I don't really understand how in this scenario both the retainedVersion and the reconciledVersion is still the range. In other scenarios I've come across the retainedVersion will still contain the range, but not the reconciledVersion.

This leaves me with a couple thoughts.

1. I can dig into Coursier and try to understand what's happening here, but this is pretty time consuming
2. I try to create a valid PURL out of the reconciled version and if it's not valid, just warn and throw it out

For the short term I'm leaning on just trying to achieve 2. This would at least guarantee that something is being submitted and also avoid odd edge cases like this.

[chrsmutti]

I'm really out of my league here as I don't have that much knowledge on dependency resolution, but coming from an user's perspective an error/warning when the PURL is invalid seems like a good idea because the only information I have at the moment is this:

All that said, thank you very much for the quick answers!

In this case in particular, I think I'm at fault as well, the upgraded dependency on the minimalDep is not on the original sbt project I was migrating, it should also depend on jackson-*:2.12.3, which would mean I wouldn't have caught this in the first place 😅

[ckipp01]

In this case in particular, I think I'm at fault as well, the upgraded dependency on the minimalDep is not on the original sbt project I was migrating, it should also depend on jackson-*:2.12.3, which would mean I wouldn't have caught this in the first place 😅

Ha, that's ok. These edge cases are helpful and hard to think of when first building something. So it's helpful see what people are hitting on.

[ckipp01]

pfff, they say PURL doesn't support ranges but you can still give one to their java library and end up with

 "package_url": "pkg:maven/com.fasterxml.jackson.core/jackson-core@%5B2.7.0%2C2.12.3%5D"

Awesome.

[ckipp01]

Alright, everything is updated @chrsmutti. If you're using v1 of the action you should be able to just try again and it will pull all the new stuff in. If that doesn't seem to, then just directly set the plugin-version in the GitHub action to use the latest 0.1.1.