Troubleshooting a failure

You can ask for help with any of this in https://gitter.im/scala/community-builds

When is this page applicable? Who is it for?

The advice on this page is about what to do when a particular project is failing because of a real issue or incompatibility in the code or a possible regression in Scala.

It's aimed not just at community build maintainers, but at external library maintainers and contributors who aren't really familiar with the community build but want to help diagnose and fix a failure.

More general information is on Maintenance; there is some overlap between that page and this.

Get oriented

Suppose a project is failing and the problem is a compilation failure or test failure that isn't the result of a missing dependency.

To initially get oriented:

• find the project's entry in configs/community.dbuild and read it, there may be useful comments, and it may be useful to know how we've customized the project's build, if it all
• also look at configs/projects.conf to see what branch or tag or whatever we’re using
  • if the community build is currently frozen, you'll only see a SHA rather than branch or tag information. in this case, consult the git history to see what branch or tag was being used before the freeze happened

Choose a troubleshooting method

Now you have a choice: do your work in dbuild, or outside dbuild?

Outside may be easier and more natural for you if you know nothing about dbuild and prefer to avoid learning it if possible, but has drawbacks.

Option 1: Outside of dbuild

This is a workable option if:

• The Scala version involved is in a binary compatible series (for example, 2.12.x nightly builds are numbered 2.12.n-bin-abcd123, the -bin- indicates binary compatibility)
• AND the version of all the dependencies involved are similar enough to what the community build uses

And may also be workable if:

• The Scala version isn't binary compatible (for example, 2.13.x nightly builds are numbered 2.13.n-pre-abcd123, the -pre- indicates a non-binary-compatible prerelease)
• AND the project in question has only one or two dependencies
• AND publishing those dependencies locally doesn't involve weird hassles
  • (you might not be able to determine this in advance, but the dependencies' config/community.dbuild entries might provide clues. otherwise just guess, and/or just try it)

If you go this route, you publish those dependencies locally for the Scala version you’re working with, and you’re good to go, it's normal development work otherwise. https://stackoverflow.com/questions/40622878/how-do-i-tell-sbt-to-use-a-nightly-build-of-scala-2-12-or-2-13 explains how to use a nightly Scala build during publishing and troubleshooting.

If you're unable to reproduce the problem the community build sees, or if local publishing proves too much hassle, you'll have to switch to the dbuild route.

Option 2: Within dbuild

Choose this option if you're already comfortable with dbuild, or if option 1 doesn't work out, or if the Scala version is binary incompatible and there are more than one or two dependencies involved.

Local runs explains how to run the community build locally. In short, you just ./run.sh or e.g. version=2.13.0-pre-abcd123 ./run.sh if the Scala version hardcoded near the top of run.sh isn't the one you want.

Saving time on dependency extraction

The first time you run the build for a particular Scala version, it takes at least an hour to extract all dependencies from all projects, so you might want to start that ahead of time and then go make yourself a sandwich or do other work.

But also, you can really save yourself some time by taking a few minutes to delete all the entries from configs/community.dbuild that you aren’t using.

configs/community.dbuild is supposed to be in an order (and usually, actually is in an order) such that you can delete all entries below any project and it’s fine and that project will still build.

That might cut down the extraction time enough all by itself that you can ./run.sh myproject and have reasonable turnaround time.

But if you want to cut the number of projects down even further in order to shorten turnaround, consult the last run log on Jenkins (or your last local run log) to see what a project’s dependencies are.

For example for Akka the log has:

[info] Project akka
[info]   depends on: cloc-plugin, genjavadoc, scala, scala-collection-compat, scala-java8-compat, scala-xml, scalacheck, scalatest

This is everything the akka entry needs, transitively, so you can keep only those entries in community.dbuild and extraction will be quite fast, especially after the first run.

Actually doing the troubleshooting

If you need to make changes to a project’s code, then fork the project and push your changes to a branch in your fork, and change config/project-refs.conf to point to your fork, and ./run.sh myproject.

Another possibility is to use dbuild’s feature where you can ask dbuild to run, then pause and open sbt for you in the project you care about, using all of the actual settings and dependencies in the actual context of the build. This feature is documented at http://lightbend.github.io/dbuild/0.9.16/introduction.html#what-if-the-build-fails . There isn't much to know, you just issue the magic dbuild checkout command with the right arguments.

I don’t generally use dbuild checkout for “normal” work, but for difficult debugging/troubleshooting, it can be a lifesaver, it cuts your turnaround time on code changes (as opposed to build config changes) down to almost nothing.

(OPTIONAL) Saving time with a local Artifactory

Extracting dependencies from lots of projects, and resolving and retrieving build-time dependencies such as sbt plugins from lots of projects, is inherently slow. If you do a lot of community build work, you'll want to run a local Artifactory to speed it up substantially, especially if your internet connection is slow.

This option is currently under-documented, but here is some advice:

• the Artifactory config to use is artifactory/artifactory-lightbend.xml
• that config file is known to work with Artifactory 6.8.2, consider using that exact version (e.g. using brew pin)
• on MacOS, installing Artifactory using brew works fine

Here's a sample .sbt/repositories file:

  local
  cache-ivy: http://127.0.0.1:8081/artifactory/scala-ci-virtual/, [organisation]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  cache: http://127.0.0.1:8081/artifactory/scala-ci-virtual/```

To require all resolution to go through Artifactory, you also want `SBT_OPTS=-Dsbt.override.build.repos=true`.

You can ask for help in https://gitter.im/scala/community-builds.