Skip to content

added scala_toolchain with scalacopts support #364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Jan 4, 2018

Conversation

or-shachar
Copy link
Contributor

WIP for #170
Initial implementation of scala_toolchain
BREAKING change - when merged we must register scala toolchain by either:
A) a call to register_scala_toolchains must happen, just like in rules_go, in order to load default toolchain.
B) defining custom scala_toolchain and registering it

Open questions:

  • How to test it? given we can't alter the WORKSPACE file of rules_scala. In ideal case - we want to load the toolchain defined in //test_expect_failure/scalacopts_from_toolchain:failing_scala_toolchain.
  • Is that okay to introduce breaking change for this feature? What is the alternative.

Still need to document

@bazel-io
Copy link

Can one of the admins verify this patch?

@ittaiz
Copy link
Contributor

ittaiz commented Dec 11, 2017 via email

@ittaiz
Copy link
Contributor

ittaiz commented Dec 11, 2017

How to test it? Easy! The e2e testing lib :) I have a passing test in a stash. I'll show you tomorrow.
Re breaking change- I think it's an option but we'd like to understand if that's the only option.
Additionally I think that if we're making a breaking change we need to understand if every additional feature there will be another breaking change (i.e. Will using scala version from it break everyone again)?
@katre is there a way for a rule to register a default toolchain?

Copy link

@katre katre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks good, only one comment.

scala/BUILD Outdated

toolchain(
name = 'scala_toolchain',
toolchain_type = '@io_bazel_rules_scala//scala:scala_toolchain.bzl',
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

toolchain_type should be a label to a target (generally a target of rule toolchain_type()), not an extension.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@katre maybe the docs should contain a snippet of my_toolchain_type so that implementors can see an example?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's a good idea, I'll add it soon.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! concretely given the lack of docs, any chance for a small snippet here? so the devs can follow up on your comment

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure:

in scala/BUILD, add:

toolchain_type(name = 'toolchain_type')

Then use '@io_bazel_rules_scala//scala:toolchain_type' as the toolchain_type parameter everywhere.

WORKSPACE Outdated
@@ -80,3 +80,6 @@ filegroup(
)
"""
)

load("//scala:scala.bzl", "register_scala_toolchains")
register_scala_toolchains()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@katre is there a way to not mandate this call by every one of our users? like a default toolchain

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, no. Bazel needs to know the available toolchains, WORKSPACE is the place for this to happen.

Eventually, when fully recursive workspace loading exists, you could place the register_scala_toolchains() call inside the WORKSPACE for the rules_scala repository, and that would be loaded, but for now it has to be in the root WORKSPACE.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see.
Two questions:

  1. java_toolchain doesn't mandate this because it's on a different implementation, right?
  2. once such a toolchain is declared can more properties be added with default values so that people won't break?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Correct, java_toolchain doesn't use this mechanism (I'm adding support this month!) Once it does, java toolchains will need to be registered, with at least one being registered by default by Bazel's initialization system.

  2. Yes, you can add further properties to the scala_toolchain. You can either include a default value in the scala_toolchain rule definition (possibly with a warning), or have the rules handle missing properties, whichever makes sense for your rule implementation.

@ittaiz
Copy link
Contributor

ittaiz commented Dec 11, 2017

To summarize the thread from the comments:
We can't have a default toolchain but once we do this breaking change we can add other properties with default values so only break people once.

scala_toolchain = rule(
_scala_toolchain_impl,
attrs = {
'scalacopts': attr.string_list(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we move the scala_library and scala_compiler jars here as well? Since, ultimately, that is how we will be able to switch from 2.11 to 2.12, etc...

Also, we might have default plugins here (since scala native and scala_js use plugins).

@ittaiz
Copy link
Contributor

ittaiz commented Dec 11, 2017 via email

@or-shachar
Copy link
Contributor Author

@ittaiz @damienmg - I am a thrilled to start using the integration testing framework (https://github.com/bazelbuild/bazel-integration-testing) :-)

I tried to use it to test out scala toolchain. I first wanted to see that I can compile scala easily. I think I got close but I failed to understand how to build the WORKSPACE file within the test. I want to somehow load current codebase of io_bazel_rules_scala as external repository using local_repository annotation. But what do I put under path? Do I need to copy/link files? The build currently fail with message:

ERROR: error loading package '': Encountered error while reading extension file 'scala/scala.bzl': no such package '@io_bazel_rules_scala//scala': /private/var/tmp/_bazel_ors/d956edd8fd2ed13432bc93c4e3b899a9/bazel-sandbox/8545224450491333639/execroot/io_bazel_rules_scala/_tmp/664cbc6d835417a9336252bb55af018e/37b39d51376b3b47b0789c75ea46aada/external/io_bazel_rules_scala must be an existing directory.

@johnynek
Copy link
Contributor

test this please

import org.specs2.mutable.{Before, SpecificationWithJUnit}
import org.specs2.specification.{BeforeAll, Scope}
import scala.collection.JavaConverters._

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little concerned that we are using specs for core testing here. Prior to this, all of our tests used scalatest and we only tested specs to exercise specifically specs. Can we have one consistent way of testing (which prior to this was scalatest)?

test_run.sh Outdated
@@ -208,6 +208,10 @@ test_scala_junit_test_can_fail() {
action_should_fail test test_expect_failure/scala_junit_test:failing_test
}

test_scalaopts_from_scala_toolchain() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we aren't calling this are we?

Also, I thought the point of using the bazel testing framework was to stop adding things to this giant script. Can we not express this test as something that we can call with bazel test ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True. I must've missed it when cleaning the test in favor to the integration test framework

@or-shachar
Copy link
Contributor Author

pardon for many commits. We must squash this PR once we decide to merge it.
eventually we failed to utilize the e2e framework as it currently requires too many manual wiring of internal and external dependencies.

@damienmg , @ittaiz - we will list our challenges and open an issue on the integration repository.

The current state is okay but not optimal: the toolchain we load globally emits warnings on unused variables. The scala_library of our test adds another scalacopts that turns warnings to fatal. That's why we expect it to fail. Very implicit, but best I got until now.

Tomorrow I'll try to make the build fail on class-name length, and try to find out if there's any importance to order of scalac options and if so - how to test that I prioritised the opts given in the libraries over the toolchain.

WORKSPACE Outdated
@@ -80,3 +80,6 @@ filegroup(
)
"""
)

# we load toolchain that fails the build on unused warnings
register_toolchains("//test_expect_failure/scalacopts_from_toolchain:failing_scala_toolchain")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this need to be in the WORKSPACE? Is that the only place to register new toolchains?

Is it the case that you can only have 1 version of each toolchain in play in a given repo? Is that how it works?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to https://docs.bazel.build/versions/master/toolchains.html#overview

When a target requests a toolchain, Bazel checks the list of registered toolchains and creates a dependency from the target to the first matching toolchain it finds. To find a matching toolchain, Bazel does the following:

Looks through the registered toolchains, first from the --experimental_extra_toolchains flag, then from the registered_toolchains calls in the project's WORKSPACE file.

For each registered toolchain, Bazel performs the following checks:

a. Does the toolchain match the requested toolchain type? If not, skip it.

b. Do the toolchain's execution and target constraints match the constraints stated in the project's execution and target platforms? If yes, the toolchain is a match.

Because Bazel always selects the first matching toolchain, order the toolchains by preference if you expect the possibility of multiple matches.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can register toolchains via a flag, or in the WORKSPACE. First match is selected.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just think it is going to confuse me and others that the only registered toolchain is on a path with "expect_failure" which just does not seem correct.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree and I think this is because we couldn't utilize bazel e2e testing :(
You do understand that this is the toolchain for a test right? It's not what users are supposed to use.
@or-shachar is there one that users are supposed to use?
@johnynek maybe making the comment clearer it's for testing?
@or-shachar why not register it via a command line flag of the specific test?

scala/BUILD Outdated
)

toolchain(
name = 'scala_toolchain',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is confusing to me. Does this string just coincidentally match the load or the item on line 4? Is this a different namespace? Can we possible not have such a name collision to avoid any confusion?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are absolutely right. I see what we can do to avoid it

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just a target name, and doesn't need to match anything, and is referenced from the register_toolchains call in the WORKSPACE.

scala/scala.bzl Outdated
@@ -955,7 +960,8 @@ scala_library = rule(
attrs={
} + _implicit_deps + _common_attrs + library_attrs + _resolve_deps,
outputs=library_outputs,
fragments = ["java"]
fragments = ["java"],
toolchains = ['@io_bazel_rules_scala//scala:scala_toolchain_type']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

weird indentation here.

scala/scala.bzl Outdated
@@ -965,7 +971,8 @@ scala_library_for_plugin_bootstrapping = rule(
implementation=_scala_library_impl,
attrs= _implicit_deps + library_attrs + _resolve_deps + _common_attrs_for_plugin_bootstrapping,
outputs=library_outputs,
fragments = ["java"]
fragments = ["java"],
toolchains = ['@io_bazel_rules_scala//scala:scala_toolchain_type']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

weird indentation here.

scala/scala.bzl Outdated
@@ -975,7 +982,8 @@ scala_macro_library = rule(
"exports": attr.label_list(allow_files=False),
} + _implicit_deps + _common_attrs + _resolve_deps,
outputs= common_outputs,
fragments = ["java"]
fragments = ["java"],
toolchains = ['@io_bazel_rules_scala//scala:scala_toolchain_type']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

scala/BUILD Outdated
visibility = ["//visibility:public"]
)

toolchain_type(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is confusing to me. What is this doing? Is this declaring a new toolchain type?

Peeve: once I noticed this it really annoyed me: why does skylark use side-effecting functions with a name parameter rather than just let bindings:

scala_toolchain_type = toolchain_type(visibility = ["//visibility:public"])

ugh...

anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The syntax is a bit awkward. I agree.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this declares a new toolchain type, which is currently just a label with no other data associated.

scala/BUILD Outdated

#call scala toolchain
scala_toolchain(
name = 'scala_toolchain_impl',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a collision too. I think we can name this scala_default_toolchain_impl or something to not confused with the toolchain rule impl.

WORKSPACE Outdated
@@ -80,3 +80,6 @@ filegroup(
)
"""
)

# we load toolchain that fails the build on unused warnings
register_toolchains("//test_expect_failure/scalacopts_from_toolchain:failing_scala_toolchain")
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can register toolchains via a flag, or in the WORKSPACE. First match is selected.

scala/BUILD Outdated
visibility = ["//visibility:public"]
)

toolchain_type(
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this declares a new toolchain type, which is currently just a label with no other data associated.

scala/BUILD Outdated
)

toolchain(
name = 'scala_toolchain',
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just a target name, and doesn't need to match anything, and is referenced from the register_toolchains call in the WORKSPACE.

scala/BUILD Outdated
)

toolchain_type(
name = "scala_toolchain_type",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Just "toolchain_type" is sufficient, since "scala" is already in the package name, and it'll make your toolchain type slightly smaller.

Ie,
@io_bazel_rules_scala//scala:scala_toolchain_type
vs
@io_bazel_rules_scala//scala:toolchain_type

This will also be consistent with the go, rust, cc, and jdk toolchain types.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we change to just toolchain_type as katre suggests?

@johnynek
Copy link
Contributor

@katre question: in scala, there are multiple backends: js, native, jvm and different versions that are incompatible (2.10, 2.11, 2.12, ....)

Are toolchains a way we build for multiple targets? For instance, I want to build for both jvm and js? So, I guess my question is: could I make a macro that builds the same rule with two different toolchains?

Critically, not all scala rules can be built with scalajs because all the transitive dependencies need to be either scalajs or js (you can't depend on a jvm library in scalajs). Similarly for scala-native (it can depend only on pure scala or C libraries). How would you suggest we tackle this issue?

@katre
Copy link

katre commented Dec 15, 2017

It depends on how the results would be used. I would probably model the different result types as different platforms: scala_jvm, scala_js, scala_native_x86. Each of these would then have a constraint for the scala compiler backend:

In scala/constraints/BUILD.bazel"

constraint_setting(name = "compiler_backend")
constraint_value(name="compiler_jvm", constraint-setting=":compiler_backend")
constraint_value(name="compiler_js", constraint-setting=":compiler_backend")
constraint_value(name="compiler_native", constraint-setting=":compiler_backend")

In scala/platforms/BUILD.bazel:

platform(name="scala_jvm",
  constraint_values = ["//scala/constraints:compiler_jvm"])
platform(name="scala_js",
  constraint_values = ["//scala/constraints:compiler_js"])
platform(name="scala_native_x86",
  constraint_values = [
    "//scala/constraints:compiler_native",
    "@bazel_tools//platforms:x86_64",
  ])

And, finally, in scala/toolchains/BUILD.bazel:

scala_toolchain(name = "toolchain_jvm",
   other config)
toolchain(name="register_toolchain_jvm",
  toolchain_type = "...:toolchain_type",
  exec_compatible_with = [],
  target_compatible_with = ["//scala/constraints:compiler_jvm"],
  toolchain = ":toolchain_jvm")
# Repeat for JS and native

Then users can select which toolchain by passing "--platforms=@io_bazel_rules_scala//scala/platforms:scala_jvm" or "...//platforms:scala_js".

The rules and toolchain would need logic to tell if a target cannot be built with that toolchain, and to emit an error if the wrong source is given.

If jvm vs js vs native output isn't what the user should be selecting with the --platforms, we can work out what the right conceptual model should be.

@ittaiz
Copy link
Contributor

ittaiz commented Dec 15, 2017 via email

@katre
Copy link

katre commented Dec 15, 2017

Currently, bazel can't support that. It's on our roadmap (you'd add the scala versions as constraints in the platforms and toolchains, and specify multiple target platforms with the "--platforms" flag), but we're not ready to start investigating the work, primarily because it will cause Bazel's internal data structures to double or more in size.

@ittaiz
Copy link
Contributor

ittaiz commented Dec 15, 2017 via email

@johnynek
Copy link
Contributor

This seems mergeable (as a first step). Is that right?

@or-shachar
Copy link
Contributor Author

@johnynek - sorry for neglecting this thread. Don't merge it yet. I want to update it a bit to follow the suggestions to better naming and then we can merge. If I won't be able to do it in the next few hours I'll surely do it by this coming Tuesday

@katre
Copy link

katre commented Jan 2, 2018

I replied on #4372, take a look there.

@or-shachar
Copy link
Contributor Author

close to rerun travis :-(

@or-shachar or-shachar closed this Jan 3, 2018
@or-shachar
Copy link
Contributor Author

open to rerun travis :-(

@or-shachar or-shachar reopened this Jan 3, 2018
@bazel-io
Copy link

bazel-io commented Jan 3, 2018

Can one of the admins verify this patch?

@ittaiz
Copy link
Contributor

ittaiz commented Jan 3, 2018

test this please (ci bot)

use cli flag to load the toolchain that should fail the test
- added "default" prefix to make it clear that its default_toolchain
- moved the toolchain_type to the top level
@ittaiz
Copy link
Contributor

ittaiz commented Jan 4, 2018

Test this please

Copy link

@katre katre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great, thanks!

@or-shachar
Copy link
Contributor Author

@katre thanks a lot! :-)
@ittaiz @johnynek - will be happy for your feedback as well. I also updated the docs.
Ready to merge

@or-shachar or-shachar changed the title WIP - added scala_toolchain with scalacopts support - tests fail added scala_toolchain with scalacopts support Jan 4, 2018
@johnynek
Copy link
Contributor

johnynek commented Jan 4, 2018

👍

I'll let @ittaiz do the honors to make sure you don't have any follow up concerns.

closes #170

README.md Outdated
scala_register_toolchains()
```
[read more here](#scala_toolchain)
# register default scala toolchain
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this section? it looks like the two lines above and it doesn't seem formatted in the "view" mode

@ittaiz ittaiz merged commit bc2940d into bazel-contrib:master Jan 4, 2018
@ittaiz
Copy link
Contributor

ittaiz commented Jan 4, 2018

@or-shachar Thanks! We've been waiting a long time for this :)
As @johnynek said next up is scala sdk. @nadavwe this is a big step towards scala 2.12 support in master.
Thanks @katre for your support!

@katre
Copy link

katre commented Jan 4, 2018

Thanks to you guys for picking up the work, I'm always happy to help out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants