You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support Bzlmod and add rules_scala to bazel-central-registry #1482
Yeah you would only want to prepare the toolchain (as a new repository) inside the extension I think. Registering it must be called from the MODULE.bazel file.
So native.register_toolchains call from the extension's starlark does not work, but having the extension create @pythons_hub//:all and then MODULE.bazel calling register_toolchains("@pythons_hub//:all") worked. 👍
@mateuszkuta256 thanks for getting this started! Any progress updates or blockers?
hi, unfortunately I'm working on other things now and won't continue with this PR in the foreseeable future
I remember this migration was on hold because interfered with 'cross compilation support'
It looks like great progress was done it this area, so maybe someone can already take over the PR
I'd like to take on the task of Bzlmodifying this repo through a series of pull requests.
I've already created a (mostly) working branch in my own fork. Though I saw a couple draft pull requests here, I ended up taking a different approach and got it mostly working. In fact, I did exactly what @sluongng suggested in #1482 (comment). (I didn't notice this comment before just now—I might've read it, but not understood it at the time—but I did study rules_python and rules_go, and ended up doing exactly that.)
There are still issues I need help to address (recorded in some of the commit messages), I didn't strictly maintain WORKSPACE compatibility (fixable), and I only tested against Scala 2.13. But I can start teasing chunks out of it in a methodical fashion, to ensure that I maintain WORKSPACE and cross-version compatibility.
For an example of what it looks like from a client perspective, here's what the MODULE.bazel stanza from my local EngFlow/example repo looks like (as opposed to the current stanza, explanatory comment notwithstanding):
So if folks are game for me to do this, I'll start carving off pieces as separate pull requests, and we can resolve any outstanding problems in the process.
Reacted by Son Luong Ngoc, hovind, Lars Albertsson, Antonio, Simonas Pinevičius, Ryan R Sundberg, Zachary King, Stuart Axelbrooke, Bartosz Kosiorek and Myroslav
Just wanted to take a moment and thank @mbland for the herculean efforts here. Seriously, this is incredible work. Not only is this a massive amount of work, but the regular comms and clear status updates have been a pleasure to keep up with. I remember suffering from a misplaced sense of self-confidence sometime last year and starting my own bzlmod branch to start hacking on this, only to quickly abandon my efforts. I'm sure I couldn't have been the only one. :) Thank you, Mike, for doing what others could not!
Reacted by Simon Mavi Stewart, Lars Albertsson, Stuart Axelbrooke, kczulko, Xùdōng Yáng and Bartosz Kosiorek
Just wanted to take a moment and thank @mbland for the herculean efforts here. Seriously, this is incredible work. Not only is this a massive amount of work, but the regular comms and clear status updates have been a pleasure to keep up with. I remember suffering from a misplaced sense of self-confidence sometime last year and starting my own bzlmod branch to start hacking on this, only to quickly abandon my efforts. I'm sure I couldn't have been the only one. :) Thank you, Mike, for doing what others could not!
@rockpunk I appreciates the appreciation. I'm fortunate to have interest in the problem, my employer's encouragement to help solve it, and the support of the maintainers and everyone here. I'm certain others could've done the work, but I'm grateful to have had the support of my employer that perhaps others didn't.
@simuons asked in #1722 (comment) whether there was a better way to reuse protobuf.patch. I'd originally created copies due to paranoia over breaking Windows builds. But of course, after I switched them from copies to symlinks, everything was fine. (Plus, I'm not sure our only Windows CI job uses any of the nested modules anyway.)
While I was at it, I decided that empty WORKSPACE.bzlmod files weren't necessary anymore, given the --noenable_workspace flag in .bazelrc. After removing all of those, #1722 is now only 70 files, of which:
13 are symlinks to protoc/0001-protobuf-19679-rm-protoc-dep.patch
17 are MODULE.bazel files mirroring existing WORKSPACE files
Updated scala_deps module extension API
@simuons asked in #1722 (comment) to replace the boolean flags in scala_deps.toolchains() with individual tag classes. The thought being, each toolchain can potentially define its own set of attributes via its own tag class.
I've since changed the API from this (example from test_version/MODULE.bazel.template):
All other tag class definitions remain the same, specifically:
settings
overridden_artifact
compiler_srcjar
scalafmt
twitter_scrooge
Fixing test_bzlmod_macros.sh on Linux and Windows
I noticed that the new test/shell/test_bzlmod_macros.sh tests from #1722 were only running on macOS in CI. I learned that one does not have to escape { in Bash regular expressions on macOS, but one must on Linux and Windows (i.e., \{).
The test then ran and failed under the test_rules_scala_jdk21 and test_rules_scala_win CI jobs. The test_rules_scala_jdk21 fix was trivial. I then updated _print_error_message from test/shell/test_helper.sh to fix the mangled error message on Windows.
That left the problem with local_path_override breaking on Windows. test_bzlmod_macros.sh generated test modules containing local_path_override calls with absolute paths. When Bazel tried to create a symlink from the MSYS2 encoded absolute path, the Windows API threw an error for want of a drive letter. The solution was to pass a relative path instead.
For more details, see the commit messages from #1722.
Reproducing and fixing Linux CI issues
In case this is useful to anyone: I develop on macOS. Until now, most of my changes have been compatible with the Linux CI build. But to really reproduce and validate the above breakage, I wanted to use the same Docker image as the CI build.
It's not available on Docker Hub, so here's how I built the image locally:
Within the container, I could then cd rules_scala and do my business.
BTW, if the rules_scala working directory was created with git-new-workdir, the .git directory will contain symlinks to absolute paths. These are valid on the host, but not inside the container, so git commands will then fail with:
mbland@bbc5d99432fc:~/rules_scala-bzlmod$ git ls-files
fatal: not a git repository (or any parent up to mount point /home/mbland)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
mbland@bbc5d99432fc:~/rules_scala-bzlmod$ ls -l .git/config
lrwxr-xr-x 1 mbland dialout 52 Oct 2 2024 .git/config -> /Users/mbland/src/bazelbuild/rules_scala/.git/config
(Hopefully this'll turn up in some poor, confused person's web searches one day.)
Reproducing and fixing Windows CI issues
I have Windows 11 installed on a personal machine with Parallels Desktop, which I used to resolve the Windows breakages via Git for Windows. I'm not sure which MSYS2 environment the CI build uses, but the Git Bash program from Git for Windows seems close enough.
Side quest: Building on Windows 11 ARM64
While I could run test_bzlmod_macros.sh on my Windows ARM64 build, other tests broke.
First, _default_platform() from protoc/private/protoc_toolchains.bzl failed. No protoc/private/protoc_integrity entry contains an exec_compat matching ["@platforms//os:windows", "@platforms//cpu:aarch64"], since no Windows ARM64 protobuf build currently exists. However, Windows ARM can run x86 binaries. So at some point, I may try to commit the following update to _default_platform():
diff --git a/protoc/private/protoc_toolchains.bzl b/protoc/private/protoc_toolchains.bzl
index 389d88c4..efbe8bf6 100644
--- a/protoc/private/protoc_toolchains.bzl+++ b/protoc/private/protoc_toolchains.bzl@@ -17,6 +17,11 @@ def _default_platform():
for platform, info in PROTOC_BUILDS.items():
if sorted(info["exec_compat"]) == host_platform:
return platform
++ # Temporary measure until native Windows ARM64 builds exist.+ if host_platform == ["@platforms//cpu:aarch64", "@platforms//os:windows"]:+ return "win64"+
fail(
"no protoc build found for host platform with constraints: " +
", ".join(HOST_CONSTRAINTS),
$ bazel build //src/...
ERROR: C:/users/msb/_bazel_msb/flxik4tq/external/bazel_tools/tools/jdk/BUILD:134:10:
While resolving toolchains for target @@bazel_tools//tools/jdk:proguard_whitelister (ada0281):
No matching toolchains found for types:
@@bazel_tools//tools/python:toolchain_type
These are branches containing changes I'll submit as pull requests after#1722 lands. All of the following are orthogonal to one another and can land in any order:
bzlmod-bumps: Updates to Bazel 7.6.1, rules_python 1.3.0, Guava 33.4.8-jre, Go 1.24.2, etc.
bzlmod-workflows: Uses the [Publish to BCR][] reusable GitHub workflow. Updates .github/workflows/release.yml and adds publish-to-bcr.yml for publishing to the [Bazel Central Registry][].
replace-scalafmt-default-repo: Removes the scalafmt_config() macro and replaces it with the new @rules_scala_toolchains//scalafmt:config target. Fixes a bug discovered while writing test_dependency_versions.sh (added in the test-dependency-versions branch described below).
minimum-dependency-versions: Separates the latest dependency versions that we test against from the minimum required versions for users. Documents these minimum version requirements in README.md.
remove-label-wrappers: Removes Label wrappers around target string literals where they're not strictly necessary. See "More repo name insights, plus Label usage" below.
The following branch depends on #1722, replace-scalafmt-default-repo, and minimum-dependency-versions:
test-dependency-versions: Adds test_dependency_versions.sh to validate the minimum Bazel and dependency versions required by rules_scala, with and without the precompiled protocol compiler toolchain. Extracts setup, run, and teardown helpers from test/shell/test_bzlmod_helpers.sh into test_runner.sh for reuse by test_dependency_versions.sh. Adds a mechanism to skip tests by prefixing their name with _ to run_test_local and run_test_ci. Adds the RULES_SCALA_TEST_REGEX environment variable to test_runner.sh. Adds documentation for RULES_SCALA_TEST_{ONLY,REGEX,VERBOSE} to the header comment of test_runner.sh. Adds ./test_dependency_versions jobs to .bazelci/presubmit.yml.
Preview branches
As always, if you want to try the functionality before #1722 lands, please only use either my bzlmod or bzlmod-bazel-8 branches. I'll keep merging all of the Pending branches into them, so they remain stable even when the pending branches change. Both contain all the changes described above.
Reacted by Atsushi Yamamoto, Michael Schuett and Stuart Axelbrooke
No, actually: Praise @simuons and @liucijus! 🙂 Not just for landing #1722, but for following up so quickly on all the refinements I've sent after that.
Now we're down to just three pull requests, all orthogonal to one another, and no pending branches! As noted below:
As always, if you want to try everything out before the next release, please only use either my bzlmod or bzlmod-bazel-8 branches. Both contain all the changes from the open pull requests.
Updated scala_toolchains() and scala_deps module extension APIs
I've added what I call "toolchains options support" in #1730. This means that scala_toolchains() accepts either boolean or dict arguments for specific toolchains, and the scala_deps module extension generates them from tag classes. This yields similar changes in both the WORKSPACE and Bzlmod APIs, specifically:
The scala_proto and twitter_scrooge tag class APIs haven't changed, but now they and the scalafmt tag class share a common implementation.
All these toolchains options dicts are aligned with the underlying toolchain macro arguments, making the relationship between them clear.
Thanks to the shared implementation, it will be easy to add options support to additional toolchains over time. These updates will require only a minor release, not a major one.
For example, under WORKSPACE, enabling scalafmt and setting a custom config changed like this:
# Beforescala_toolchains(
# Other toolchains settings...scalafmt=True,
scalafmt_default_config="//path/to/my/custom:scalafmt.conf",
)
# Afterscala_toolchains(
# Other toolchains settings...scalafmt= {"default_config": "//path/to/my/custom:scalafmt.conf"},
)
scalafmt = True will still work if you want to use the default options, but it's implied if you supply your own options object.
Under Bzlmod, the scala_proto tag class changed thus to align the interface with setup_scala_proto_toolchains from //scala_proto:toolchains.bzl:
scala_deps.scala_proto() will work in this case, since under the hood, {"default_gen_opts": ["grpc"]} is the default.
Please see the description of #1730 for further details on all the good things.
This came about after @simuons asked in #1722 (comment) to replace the boolean flags in scala_deps.toolchains() with individual tag classes. The thought being, each toolchain can potentially define its own set of attributes via its own tag class. (I mentioned all this in last week's update.)
In the same comment, he also asked whether we could replace toolchain macros (invoked by users in BUILD files) entirely with a module extension. Though I noted this won't be trivial to do, and shouldn't hold up the release, I later experimented with the toolchain macro replacement idea. I'm not yet near achieving that goal, but my experimentation yielded the immediate improvements described above and in #1730.
We're down to one last pull request (for real, this time!): #1730.
With this last API change (described in last week's update), pushing the v7.0.0 tag should publish the rules_scala module to the Bazel Central Registry. This is because #1731 put the necessary Publish to BCR configuration into place. Future dependency version bumps and backward compatible API changes can happen in minor version updates easily after that.
Ideally, #1729 would make it in before publishing v7.0.0 as well, but it's not strictly required. It adds tests to enforce the minimum dependency version guarantees from #1726, but it's not technically blocking v7.0.0. (Same with #1732, a routine dependency version bump.)
As always, if you want to try everything out before the next release, please only use either my bzlmod or bzlmod-bazel-8 branches. Both contain all the changes from the open pull requests.
Thanks to @meteorcloudy, the repo has moved from bazelbuild/rules_scala to bazel-contrib/rules_scala per #1616. The lack of Bzlmod support was the last blocker. Now that #1722 is in, and publishing the first rules_scala module is imminent, moving to bazel-contrib now streamlines the publishing process.
Toolchains blog post
I'm now working in earnest on the next installment in my Bzlmod blog post series, based on this rules_scala experience. I don't know for sure yet when it'll be ready, but I'm having fun retracing all the steps. If I can manage to explain them to myself, hopefully the result will be useful to others, too.
Module extension toolchains experiment
To largely repeat #1730 (comment): I've continued to experiment with generating toolchains from the scala_deps module extension in the bzlmod-toolchains-experiment branch of mbland/rules_scala. I've managed to generate a separate toolchain in @rules_scala_toolchains//scala after updating the scala_deps.scala() tag class.
This confirms that, after #1730 lands, and after the new release, we can expand the tag classes to pass through more parameters. It also confirms that we can generate new toolchains in @rules_scala_toolchains using a module extension.
Such updates can remain backward compatible, requiring only minor version releases. It may even be possible to support an arbitrary number of toolchains and/or separate toolchain repos in a minor version update. I've yet to get around to experimenting with that yet, however.
The goal is to follow through on @simuons's idea of configuring all toolchains via module extensions from #1722 (comment). It's not worth blocking v7.0.0, as existing setup_scala_toolchain macro calls from BUILD files will continue to work. But it might be more idiomatic, following toolchain configuration patterns like those in rules_go and rules_python. (I studied those two rule sets to develop scala_toolchains() and the scala_deps extension in the first place.)
If/when I reach a point where replacing BUILD macros with module extensions is a possibility, I don't think it'll require a major version release. We can publish the new mechanism in a minor release and see how well it works in practice. Then, if we want to make the toolchain setup macros private, that can happen in v8.0.0 or some other future major version release.
Reacted by Xùdōng Yáng, Jeff Klukas, Simonas Pinevičius and Yun Peng
So @simuons and @liucijus, I suppose it's up to you to publish v7.0.0 when you like. The only thing is you may consider creating your own Personal Access Token to publish to the BCR, per the dicsussion in #1731.
Hey @simuons@liucijus
I created https://github.com/bazel-contrib-bot, granted it write access to the bazel-contrib/bazel-central-registry fork, and added its PAT at the org level.
So you don't have to do anything. Just push the v7.0.0 tag to the repo (or let me know if you'd like me to release)
Release went great. Only mistake is the BCR PR authored by Aspect's bot, due to a repo PAT that shadowed the org PAT. I've removed the former, so next time this runs it will be authored by @bazel-contrib-bot.
I'm adding commits on the PR to the BCR to green it up.
P.S. I've published Migrating to Bazel Modules (a.k.a. Bzlmod) - Toolchainization, which describes the design of the @rules_scala_toolchains repo, scala_toolchains macro, and scala_deps module extension in detail. It also talks a little about @rules_scala_protoc_toolchains as well.
Other blog posts in the series also draw examples from my work on this issue, particularly with regard to handling repository names properly. But this post tries to make accessible perhaps the most esoteric change of the whole undertaking.
Activity
chrislovecnm commentedon Jul 11, 2023
Just a note to anyone that starts the work
native.register_toolchainsis not supported when using bzlmod.sluongng commentedon Jul 11, 2023
@chrislovecnm it's still available here no? https://bazel.build/rules/lib/globals/module#register_toolchains
chrislovecnm commentedon Jul 11, 2023
@sluongng When I last tried it, it did not work when using it inside an extension. In rules_python we are skipping the native call:
https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/python/repositories.bzl#L569
And build the toolchains here:
https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/python/extensions/private/pythons_hub.bzl#L56
We build the toolchains and use the following:
https://github.com/bazelbuild/rules_python/blob/36082079b514529d4009a2b104475fde1cdd5b30/MODULE.bazel#L49
sluongng commentedon Jul 11, 2023
Yeah you would only want to prepare the toolchain (as a new repository) inside the extension I think. Registering it must be called from the MODULE.bazel file.
So
native.register_toolchainscall from the extension's starlark does not work, but having the extension create@pythons_hub//:alland then MODULE.bazel callingregister_toolchains("@pythons_hub//:all")worked. 👍mateuszkuta256 commentedon Jan 24, 2024
I prepared a minimal bzlmod support here and gonna split it into a few PRs
TODOs after merging it:
SCALA_VERSIONwill probably be configurable via repo-env)pcj commentedon Jun 30, 2024
@mateuszkuta256 thanks for getting this started! Any progress updates or blockers?
mateuszkuta256 commentedon Jul 1, 2024
hi, unfortunately I'm working on other things now and won't continue with this PR in the foreseeable future
I remember this migration was on hold because interfered with 'cross compilation support'
It looks like great progress was done it this area, so maybe someone can already take over the PR
mbland commentedon Oct 2, 2024
I'd like to take on the task of Bzlmodifying this repo through a series of pull requests.
I've already created a (mostly) working branch in my own fork. Though I saw a couple draft pull requests here, I ended up taking a different approach and got it mostly working. In fact, I did exactly what @sluongng suggested in #1482 (comment). (I didn't notice this comment before just now—I might've read it, but not understood it at the time—but I did study rules_python and rules_go, and ended up doing exactly that.)
There are still issues I need help to address (recorded in some of the commit messages), I didn't strictly maintain
WORKSPACEcompatibility (fixable), and I only tested against Scala 2.13. But I can start teasing chunks out of it in a methodical fashion, to ensure that I maintainWORKSPACEand cross-version compatibility.For an example of what it looks like from a client perspective, here's what the
MODULE.bazelstanza from my local EngFlow/example repo looks like (as opposed to the current stanza, explanatory comment notwithstanding):So if folks are game for me to do this, I'll start carving off pieces as separate pull requests, and we can resolve any outstanding problems in the process.
cc: @BillyAutrey @jayconrod @benjaminp @TheGrizzlyDev
Bump Bazel and dependencies to prepare for Bzlmod
Fix Bazel 7 related protobuf build failures
Fix Bazel 7 related protobuf build failures
Fix Bazel 7 related protobuf build failures
Fix Bazel 7 related protobuf build failures
Add apparent_repo_name macro to prepare for Bzlmod
374 remaining items
rockpunk commentedon Apr 20, 2025
Just wanted to take a moment and thank @mbland for the herculean efforts here. Seriously, this is incredible work. Not only is this a massive amount of work, but the regular comms and clear status updates have been a pleasure to keep up with. I remember suffering from a misplaced sense of self-confidence sometime last year and starting my own bzlmod branch to start hacking on this, only to quickly abandon my efforts. I'm sure I couldn't have been the only one. :) Thank you, Mike, for doing what others could not!
mbland commentedon Apr 22, 2025
@rockpunk I appreciates the appreciation. I'm fortunate to have interest in the problem, my employer's encouragement to help solve it, and the support of the maintainers and everyone here. I'm certain others could've done the work, but I'm grateful to have had the support of my employer that perhaps others didn't.
Of course, the work, ever mysterious and important, isn't quite over yet, which brings me to the next...
Weekly update! (And I'm glad to hear someone other than me gets value out of my weekly brain dumps.)
Overview
#1722 has changed a little bit, as described in the sections below.
All pending change branches are up to date with the latest in #1722, as are the bzlmod and bzlmod-bazel-8 branches.
Open
Repo names blog post
As promised, I published Migrating to Bazel Modules (a.k.a. Bzlmod) - Repo Names, Again..., using pull requests from this effort as examples. I also posted an announcement of this post in my blog, which contains a suitable reference that I'd missed in the original post.
Removing unnecessary artifacts
@simuons asked in #1722 (comment) whether there was a better way to reuse
protobuf.patch. I'd originally created copies due to paranoia over breaking Windows builds. But of course, after I switched them from copies to symlinks, everything was fine. (Plus, I'm not sure our only Windows CI job uses any of the nested modules anyway.)While I was at it, I decided that empty
WORKSPACE.bzlmodfiles weren't necessary anymore, given the--noenable_workspaceflag in.bazelrc. After removing all of those, #1722 is now only 70 files, of which:protoc/0001-protobuf-19679-rm-protoc-dep.patchMODULE.bazelfiles mirroring existingWORKSPACEfilesUpdated
scala_depsmodule extension API@simuons asked in #1722 (comment) to replace the boolean flags in
scala_deps.toolchains()with individual tag classes. The thought being, each toolchain can potentially define its own set of attributes via its own tag class.I've since changed the API from this (example from
test_version/MODULE.bazel.template):to this:
Note that:
scala_deps.toolchains(toolchain = True)in the previous API.scala_deps.scala(), whereas the previous API defaulted toscala_deps.toolchains(scala = True).scala_deps.scala_proto(options = [...])replacesscala_deps.toolchains(scala_proto_options = [...]).All other tag class definitions remain the same, specifically:
settingsoverridden_artifactcompiler_srcjarscalafmttwitter_scroogeFixing
test_bzlmod_macros.shon Linux and WindowsI noticed that the new
test/shell/test_bzlmod_macros.shtests from #1722 were only running on macOS in CI. I learned that one does not have to escape{in Bash regular expressions on macOS, but one must on Linux and Windows (i.e.,\{).The test then ran and failed under the
test_rules_scala_jdk21andtest_rules_scala_winCI jobs. Thetest_rules_scala_jdk21fix was trivial. I then updated_print_error_messagefromtest/shell/test_helper.shto fix the mangled error message on Windows.That left the problem with local_path_override breaking on Windows.
test_bzlmod_macros.shgenerated test modules containinglocal_path_overridecalls with absolute paths. When Bazel tried to create a symlink from the MSYS2 encoded absolute path, the Windows API threw an error for want of a drive letter. The solution was to pass a relative path instead.For more details, see the commit messages from #1722.
Reproducing and fixing Linux CI issues
In case this is useful to anyone: I develop on macOS. Until now, most of my changes have been compatible with the Linux CI build. But to really reproduce and validate the above breakage, I wanted to use the same Docker image as the CI build.
It's not available on Docker Hub, so here's how I built the image locally:
Here's
ubuntu2004.patch:Then I would run the following script within the
rules_scaladirectory:Within the container, I could then
cd rules_scalaand do my business.BTW, if the
rules_scalaworking directory was created with git-new-workdir, the.gitdirectory will contain symlinks to absolute paths. These are valid on the host, but not inside the container, sogitcommands will then fail with:(Hopefully this'll turn up in some poor, confused person's web searches one day.)
Reproducing and fixing Windows CI issues
I have Windows 11 installed on a personal machine with Parallels Desktop, which I used to resolve the Windows breakages via Git for Windows. I'm not sure which MSYS2 environment the CI build uses, but the Git Bash program from Git for Windows seems close enough.
Side quest: Building on Windows 11 ARM64
While I could run
test_bzlmod_macros.shon my Windows ARM64 build, other tests broke.First,
_default_platform()fromprotoc/private/protoc_toolchains.bzlfailed. Noprotoc/private/protoc_integrityentry contains anexec_compatmatching["@platforms//os:windows", "@platforms//cpu:aarch64"], since no Windows ARM64protobufbuild currently exists. However, Windows ARM can run x86 binaries. So at some point, I may try to commit the following update to_default_platform():(Note: The
string.joinin the last line is a bug fix from minimum-dependency-versions.)The trickier problem is
rules_python:Note that @rules_python//python:toolchain_type is an alias to @bazel_tools//tools/python:toolchain_type.
However, Windows ARM64 support may be forthcoming, given recent movement on:
Pending branches
These are branches containing changes I'll submit as pull requests after #1722 lands. All of the following are orthogonal to one another and can land in any order:
bzlmod-bumps: Updates to Bazel 7.6.1, rules_python 1.3.0, Guava 33.4.8-jre, Go 1.24.2, etc.
test-src: Adds
src/...to thebazel buildandbazel testcommands intest_rules_scala.shto follow up on Add Java 24 compatibility #1719 and Cleanup WorkerTest after SecurityManager was removed #1721. Joins thecontentslines inWorkerTest.testPersistentWorkerArgsfileusing theline.separatorsystem property to fix a test failure on Windows.bzlmod-workflows: Uses the [Publish to BCR][] reusable GitHub workflow. Updates
.github/workflows/release.ymland addspublish-to-bcr.ymlfor publishing to the [Bazel Central Registry][].replace-scalafmt-default-repo: Removes the
scalafmt_config()macro and replaces it with the new@rules_scala_toolchains//scalafmt:configtarget. Fixes a bug discovered while writingtest_dependency_versions.sh(added in thetest-dependency-versionsbranch described below).minimum-dependency-versions: Separates the latest dependency versions that we test against from the minimum required versions for users. Documents these minimum version requirements in
README.md.remove-label-wrappers: Removes
Labelwrappers around target string literals where they're not strictly necessary. See "More repo name insights, plusLabelusage" below.The following branch depends on #1722,
replace-scalafmt-default-repo, andminimum-dependency-versions:test_dependency_versions.shto validate the minimum Bazel and dependency versions required byrules_scala, with and without the precompiled protocol compiler toolchain. Extracts setup, run, and teardown helpers fromtest/shell/test_bzlmod_helpers.shintotest_runner.shfor reuse bytest_dependency_versions.sh. Adds a mechanism to skip tests by prefixing their name with_torun_test_localandrun_test_ci. Adds theRULES_SCALA_TEST_REGEXenvironment variable totest_runner.sh. Adds documentation forRULES_SCALA_TEST_{ONLY,REGEX,VERBOSE}to the header comment oftest_runner.sh. Adds./test_dependency_versionsjobs to.bazelci/presubmit.yml.Preview branches
As always, if you want to try the functionality before #1722 lands, please only use either my bzlmod or bzlmod-bazel-8 branches. I'll keep merging all of the Pending branches into them, so they remain stable even when the pending branches change. Both contain all the changes described above.
mbland commentedon Apr 26, 2025
D'oh! Didn't mean for #1722 to close this yet. I guess my commit/PR message must've triggered that.
@simuons @liucijus Do you want to reopen this until we publish the module, or open a new issue?
mbland commentedon Apr 29, 2025
Weekly update!
Overview
#1722 is in, and MODULE.bazel exists on
master. Praise Kier!No, actually: Praise @simuons and @liucijus! 🙂 Not just for landing #1722, but for following up so quickly on all the refinements I've sent after that.
Now we're down to just three pull requests, all orthogonal to one another, and no pending branches! As noted below:
WORKSPACEand Bzlmod.Once those two land (plus maybe #1729), we're ready to release and publish this thing and close out here.
Merged
@scalafmt_defaultwith toolchains target #1725Labelwrappers for target string literals #1728Open
Pending branches
None! That's it! No more!
Preview branches
As always, if you want to try everything out before the next release, please only use either my bzlmod or bzlmod-bazel-8 branches. Both contain all the changes from the open pull requests.
Updated
scala_toolchains()andscala_depsmodule extension APIsI've added what I call "toolchains options support" in #1730. This means that
scala_toolchains()accepts either boolean or dict arguments for specific toolchains, and thescala_depsmodule extension generates them from tag classes. This yields similar changes in both theWORKSPACEand Bzlmod APIs, specifically:Add toolchain options API to WORKSPACE and Bzlmod #1730 adds toolchain options support to the
scala_toolchains()parametersscalafmt,scala_proto, andtwitter_scrooge, and to thescalafmttag class.Add toolchain options API to WORKSPACE and Bzlmod #1730 eliminates the
scalafmt_default_config,scala_proto_options, andtwitter_scrooge_depsoption parameters fromscala_toolchains().The
scala_protoandtwitter_scroogetag class APIs haven't changed, but now they and thescalafmttag class share a common implementation.All these toolchains options dicts are aligned with the underlying toolchain macro arguments, making the relationship between them clear.
Thanks to the shared implementation, it will be easy to add options support to additional toolchains over time. These updates will require only a minor release, not a major one.
For example, under
WORKSPACE, enablingscalafmtand setting a custom config changed like this:scalafmt = Truewill still work if you want to use the default options, but it's implied if you supply your own options object.Under Bzlmod, the
scala_prototag class changed thus to align the interface withsetup_scala_proto_toolchainsfrom//scala_proto:toolchains.bzl:scala_deps.scala_proto()will work in this case, since under the hood,{"default_gen_opts": ["grpc"]}is the default.Please see the description of #1730 for further details on all the good things.
This came about after @simuons asked in #1722 (comment) to replace the boolean flags in
scala_deps.toolchains()with individual tag classes. The thought being, each toolchain can potentially define its own set of attributes via its own tag class. (I mentioned all this in last week's update.)In the same comment, he also asked whether we could replace toolchain macros (invoked by users in
BUILDfiles) entirely with a module extension. Though I noted this won't be trivial to do, and shouldn't hold up the release, I later experimented with the toolchain macro replacement idea. I'm not yet near achieving that goal, but my experimentation yielded the immediate improvements described above and in #1730.mbland commentedon May 6, 2025
Weekly update!
Overview
We're down to one last pull request (for real, this time!): #1730.
With this last API change (described in last week's update), pushing the
v7.0.0tag should publish therules_scalamodule to the Bazel Central Registry. This is because #1731 put the necessary Publish to BCR configuration into place. Future dependency version bumps and backward compatible API changes can happen in minor version updates easily after that.Ideally, #1729 would make it in before publishing v7.0.0 as well, but it's not strictly required. It adds tests to enforce the minimum dependency version guarantees from #1726, but it's not technically blocking v7.0.0. (Same with #1732, a routine dependency version bump.)
Merged
Open
Preview branches
As always, if you want to try everything out before the next release, please only use either my bzlmod or bzlmod-bazel-8 branches. Both contain all the changes from the open pull requests.
Move to the bazel-contrib GitHub org
Thanks to @meteorcloudy, the repo has moved from bazelbuild/rules_scala to bazel-contrib/rules_scala per #1616. The lack of Bzlmod support was the last blocker. Now that #1722 is in, and publishing the first
rules_scalamodule is imminent, moving to bazel-contrib now streamlines the publishing process.Toolchains blog post
I'm now working in earnest on the next installment in my Bzlmod blog post series, based on this
rules_scalaexperience. I don't know for sure yet when it'll be ready, but I'm having fun retracing all the steps. If I can manage to explain them to myself, hopefully the result will be useful to others, too.Module extension toolchains experiment
To largely repeat #1730 (comment): I've continued to experiment with generating toolchains from the
scala_depsmodule extension in the bzlmod-toolchains-experiment branch of mbland/rules_scala. I've managed to generate a separate toolchain in@rules_scala_toolchains//scalaafter updating thescala_deps.scala()tag class.This confirms that, after #1730 lands, and after the new release, we can expand the tag classes to pass through more parameters. It also confirms that we can generate new toolchains in
@rules_scala_toolchainsusing a module extension.Such updates can remain backward compatible, requiring only minor version releases. It may even be possible to support an arbitrary number of toolchains and/or separate toolchain repos in a minor version update. I've yet to get around to experimenting with that yet, however.
The goal is to follow through on @simuons's idea of configuring all toolchains via module extensions from #1722 (comment). It's not worth blocking v7.0.0, as existing
setup_scala_toolchainmacro calls fromBUILDfiles will continue to work. But it might be more idiomatic, following toolchain configuration patterns like those inrules_goandrules_python. (I studied those two rule sets to developscala_toolchains()and thescala_depsextension in the first place.)If/when I reach a point where replacing
BUILDmacros with module extensions is a possibility, I don't think it'll require a major version release. We can publish the new mechanism in a minor release and see how well it works in practice. Then, if we want to make the toolchain setup macros private, that can happen in v8.0.0 or some other future major version release.mbland commentedon May 7, 2025
Daily update! 😛
@simuons merged #1729, #1730, and #1732 today, as well as @WojciechMazur's #1733.
So @simuons and @liucijus, I suppose it's up to you to publish v7.0.0 when you like. The only thing is you may consider creating your own Personal Access Token to publish to the BCR, per the dicsussion in #1731.
Then once https://registry.bazel.build/modules/rules_scala goes from HTTP 404 to HTTP 200, we're done here! 😄
In other news, I did just open #1734 with a very small documentation update (on a tip from @gergelyfabian); nothing that should block the release.
I'll continue to experiment with my bzlmod-toolchains-experiment branch for post-v7.0.0 releases.
alexeagle commentedon May 8, 2025
Hey @simuons @liucijus
I created https://github.com/bazel-contrib-bot, granted it write access to the bazel-contrib/bazel-central-registry fork, and added its PAT at the org level.
So you don't have to do anything. Just push the
v7.0.0tag to the repo (or let me know if you'd like me to release)simuons commentedon May 9, 2025
Tried to push
v7.0.0for a release and workflow failed https://github.com/bazel-contrib/rules_scala/actions/runs/14926599712Opened a PR attempting to fix this #1736
@alexeagle would appreciate if you could take a look.
alexeagle commentedon May 9, 2025
I'll update the tag and retry, in case something else is broken I can do the quick fixes
alexeagle commentedon May 9, 2025
Release went great. Only mistake is the BCR PR authored by Aspect's bot, due to a repo PAT that shadowed the org PAT. I've removed the former, so next time this runs it will be authored by @bazel-contrib-bot.
I'm adding commits on the PR to the BCR to green it up.
https://github.com/bazelbuild/bazel-central-registry/pull/4552/commits
so I'll disable windows testing for this release
--incompatible_autoload_externally=- https://buildkite.com/bazel/bcr-presubmit/builds/13836#0196b5c8-6a5c-475f-baf3-8259aacd5744this can be remediated by adding a comment on the PR
mbland commentedon May 9, 2025
So bazelbuild/bazel-central-registry#4552 is passing. We're just waiting on a maintainer to merge it so the module will appear on the Bazel Central Registry at https://registry.bazel.build/modules/rules_scala.
In the meanwhile, you can grab v7.0.0 from GitHub right now. If you're still a
WORKSPACEuser, well, there you go. Have at it.If you're an impatient Bzlmod user, and can't wait another day or so for the BCR module, you can do like I just did in EngFlow/example and use it right now via archive_override:
Thanks much to @simuons and @alexeagle for getting v7.0.0 out today, and getting us to this one final step away from finishing this thing!
mbland commentedon May 9, 2025
And now, thanks to @Wyverald, you can go straight to
bazel_dep(name = "rules_scala", version = "7.0.0")now. 😄@simuons and @liucijus: Ready to close this?
simuons commentedon May 12, 2025
@mbland thank you so much for resolving this!
mbland commentedon May 12, 2025
@simuons My pleasure! Couldn't've done it without all your help and support, and that of so many others who participated in this process. 😄
mbland commentedon May 14, 2025
P.S. I've published Migrating to Bazel Modules (a.k.a. Bzlmod) - Toolchainization, which describes the design of the
@rules_scala_toolchainsrepo,scala_toolchainsmacro, andscala_depsmodule extension in detail. It also talks a little about@rules_scala_protoc_toolchainsas well.Other blog posts in the series also draw examples from my work on this issue, particularly with regard to handling repository names properly. But this post tries to make accessible perhaps the most esoteric change of the whole undertaking.