-
-
Notifications
You must be signed in to change notification settings - Fork 286
added scala_toolchain with scalacopts support #364
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Can one of the admins verify this patch? |
Test this please
…On Mon, 11 Dec 2017 at 18:16 bazel.io machine account < ***@***.***> wrote:
Can one of the admins verify this patch?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#364 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABUIF6wCpfXk0gaDsuyBJq1YX7Wl4lKAks5s_VVggaJpZM4Q9pYk>
.
|
How to test it? Easy! The e2e testing lib :) I have a passing test in a stash. I'll show you tomorrow. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, this looks good, only one comment.
scala/BUILD
Outdated
|
||
toolchain( | ||
name = 'scala_toolchain', | ||
toolchain_type = '@io_bazel_rules_scala//scala:scala_toolchain.bzl', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
toolchain_type should be a label to a target (generally a target of rule toolchain_type()), not an extension.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that's a good idea, I'll add it soon.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks! concretely given the lack of docs, any chance for a small snippet here? so the devs can follow up on your comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sure:
in scala/BUILD, add:
toolchain_type(name = 'toolchain_type')
Then use '@io_bazel_rules_scala//scala:toolchain_type' as the toolchain_type parameter everywhere.
WORKSPACE
Outdated
@@ -80,3 +80,6 @@ filegroup( | |||
) | |||
""" | |||
) | |||
|
|||
load("//scala:scala.bzl", "register_scala_toolchains") | |||
register_scala_toolchains() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@katre is there a way to not mandate this call by every one of our users? like a default toolchain
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately, no. Bazel needs to know the available toolchains, WORKSPACE is the place for this to happen.
Eventually, when fully recursive workspace loading exists, you could place the register_scala_toolchains() call inside the WORKSPACE for the rules_scala repository, and that would be loaded, but for now it has to be in the root WORKSPACE.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see.
Two questions:
- java_toolchain doesn't mandate this because it's on a different implementation, right?
- once such a toolchain is declared can more properties be added with default values so that people won't break?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
-
Correct, java_toolchain doesn't use this mechanism (I'm adding support this month!) Once it does, java toolchains will need to be registered, with at least one being registered by default by Bazel's initialization system.
-
Yes, you can add further properties to the scala_toolchain. You can either include a default value in the scala_toolchain rule definition (possibly with a warning), or have the rules handle missing properties, whichever makes sense for your rule implementation.
To summarize the thread from the comments: |
scala_toolchain = rule( | ||
_scala_toolchain_impl, | ||
attrs = { | ||
'scalacopts': attr.string_list(), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could we move the scala_library and scala_compiler jars here as well? Since, ultimately, that is how we will be able to switch from 2.11 to 2.12, etc...
Also, we might have default plugins here (since scala native and scala_js use plugins).
Oscar,
I asked Or and Idan to start with a minimal PR to test the waters and to
chunk it down (they have ~2 more days to work on it).
I definitely agree we want the scala version to be in here and that is one
of the big motivations for this work.
We want to move very soon to 2.12 and the whole branch thing doesn't scale.
Re scala library and scala_compiler_jars- I also thought about it but now
I'm not sure about the cons. IIUC if we do the granularity of scala version
with those then if I want to have a custom toolchain for custom scalac opts
will I need to supply all the jar labels myself? I'd rather just say
2.11.7
…On Mon, Dec 11, 2017 at 8:49 PM P. Oscar Boykin ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In scala/scala_toolchain.bzl
<#364 (comment)>
:
> @@ -0,0 +1,12 @@
+def _scala_toolchain_impl(ctx):
+ toolchain = platform_common.ToolchainInfo(
+ scalacopts = ctx.attr.scalacopts,
+ )
+ return [toolchain]
+
+scala_toolchain = rule(
+ _scala_toolchain_impl,
+ attrs = {
+ 'scalacopts': attr.string_list(),
could we move the scala_library and scala_compiler jars here as well?
Since, ultimately, that is how we will be able to switch from 2.11 to 2.12,
etc...
Also, we might have default plugins here (since scala native and scala_js
use plugins).
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#364 (review)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABUIF4lncNGjUjgzwgxBxQLtRd0MKxb7ks5s_XkhgaJpZM4Q9pYk>
.
|
@ittaiz @damienmg - I am a thrilled to start using the integration testing framework (https://github.com/bazelbuild/bazel-integration-testing) :-) I tried to use it to test out scala toolchain. I first wanted to see that I can compile scala easily. I think I got close but I failed to understand how to build the
|
test this please |
import org.specs2.mutable.{Before, SpecificationWithJUnit} | ||
import org.specs2.specification.{BeforeAll, Scope} | ||
import scala.collection.JavaConverters._ | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a little concerned that we are using specs for core testing here. Prior to this, all of our tests used scalatest and we only tested specs to exercise specifically specs. Can we have one consistent way of testing (which prior to this was scalatest)?
test_run.sh
Outdated
@@ -208,6 +208,10 @@ test_scala_junit_test_can_fail() { | |||
action_should_fail test test_expect_failure/scala_junit_test:failing_test | |||
} | |||
|
|||
test_scalaopts_from_scala_toolchain() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we aren't calling this are we?
Also, I thought the point of using the bazel testing framework was to stop adding things to this giant script. Can we not express this test as something that we can call with bazel test
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
True. I must've missed it when cleaning the test in favor to the integration test framework
pardon for many commits. We must squash this PR once we decide to merge it. @damienmg , @ittaiz - we will list our challenges and open an issue on the integration repository. The current state is okay but not optimal: the toolchain we load globally emits warnings on unused variables. The Tomorrow I'll try to make the build fail on class-name length, and try to find out if there's any importance to order of scalac options and if so - how to test that I prioritised the opts given in the libraries over the toolchain. |
WORKSPACE
Outdated
@@ -80,3 +80,6 @@ filegroup( | |||
) | |||
""" | |||
) | |||
|
|||
# we load toolchain that fails the build on unused warnings | |||
register_toolchains("//test_expect_failure/scalacopts_from_toolchain:failing_scala_toolchain") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does this need to be in the WORKSPACE? Is that the only place to register new toolchains?
Is it the case that you can only have 1 version of each toolchain in play in a given repo? Is that how it works?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
according to https://docs.bazel.build/versions/master/toolchains.html#overview
When a target requests a toolchain, Bazel checks the list of registered toolchains and creates a dependency from the target to the first matching toolchain it finds. To find a matching toolchain, Bazel does the following:
Looks through the registered toolchains, first from the --experimental_extra_toolchains flag, then from the registered_toolchains calls in the project's WORKSPACE file.
For each registered toolchain, Bazel performs the following checks:
a. Does the toolchain match the requested toolchain type? If not, skip it.
b. Do the toolchain's execution and target constraints match the constraints stated in the project's execution and target platforms? If yes, the toolchain is a match.
Because Bazel always selects the first matching toolchain, order the toolchains by preference if you expect the possibility of multiple matches.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can register toolchains via a flag, or in the WORKSPACE. First match is selected.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just think it is going to confuse me and others that the only registered toolchain is on a path with "expect_failure" which just does not seem correct.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree and I think this is because we couldn't utilize bazel e2e testing :(
You do understand that this is the toolchain for a test right? It's not what users are supposed to use.
@or-shachar is there one that users are supposed to use?
@johnynek maybe making the comment clearer it's for testing?
@or-shachar why not register it via a command line flag of the specific test?
scala/BUILD
Outdated
) | ||
|
||
toolchain( | ||
name = 'scala_toolchain', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is confusing to me. Does this string just coincidentally match the load or the item on line 4? Is this a different namespace? Can we possible not have such a name collision to avoid any confusion?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are absolutely right. I see what we can do to avoid it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's just a target name, and doesn't need to match anything, and is referenced from the register_toolchains call in the WORKSPACE.
scala/scala.bzl
Outdated
@@ -955,7 +960,8 @@ scala_library = rule( | |||
attrs={ | |||
} + _implicit_deps + _common_attrs + library_attrs + _resolve_deps, | |||
outputs=library_outputs, | |||
fragments = ["java"] | |||
fragments = ["java"], | |||
toolchains = ['@io_bazel_rules_scala//scala:scala_toolchain_type'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
weird indentation here.
scala/scala.bzl
Outdated
@@ -965,7 +971,8 @@ scala_library_for_plugin_bootstrapping = rule( | |||
implementation=_scala_library_impl, | |||
attrs= _implicit_deps + library_attrs + _resolve_deps + _common_attrs_for_plugin_bootstrapping, | |||
outputs=library_outputs, | |||
fragments = ["java"] | |||
fragments = ["java"], | |||
toolchains = ['@io_bazel_rules_scala//scala:scala_toolchain_type'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
weird indentation here.
scala/scala.bzl
Outdated
@@ -975,7 +982,8 @@ scala_macro_library = rule( | |||
"exports": attr.label_list(allow_files=False), | |||
} + _implicit_deps + _common_attrs + _resolve_deps, | |||
outputs= common_outputs, | |||
fragments = ["java"] | |||
fragments = ["java"], | |||
toolchains = ['@io_bazel_rules_scala//scala:scala_toolchain_type'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto
scala/BUILD
Outdated
visibility = ["//visibility:public"] | ||
) | ||
|
||
toolchain_type( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is confusing to me. What is this doing? Is this declaring a new toolchain type?
Peeve: once I noticed this it really annoyed me: why does skylark use side-effecting functions with a name
parameter rather than just let bindings:
scala_toolchain_type = toolchain_type(visibility = ["//visibility:public"])
ugh...
anyway.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The syntax is a bit awkward. I agree.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this declares a new toolchain type, which is currently just a label with no other data associated.
scala/BUILD
Outdated
|
||
#call scala toolchain | ||
scala_toolchain( | ||
name = 'scala_toolchain_impl', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is a collision too. I think we can name this scala_default_toolchain_impl or something to not confused with the toolchain rule impl.
WORKSPACE
Outdated
@@ -80,3 +80,6 @@ filegroup( | |||
) | |||
""" | |||
) | |||
|
|||
# we load toolchain that fails the build on unused warnings | |||
register_toolchains("//test_expect_failure/scalacopts_from_toolchain:failing_scala_toolchain") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can register toolchains via a flag, or in the WORKSPACE. First match is selected.
scala/BUILD
Outdated
visibility = ["//visibility:public"] | ||
) | ||
|
||
toolchain_type( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this declares a new toolchain type, which is currently just a label with no other data associated.
scala/BUILD
Outdated
) | ||
|
||
toolchain( | ||
name = 'scala_toolchain', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's just a target name, and doesn't need to match anything, and is referenced from the register_toolchains call in the WORKSPACE.
scala/BUILD
Outdated
) | ||
|
||
toolchain_type( | ||
name = "scala_toolchain_type", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: Just "toolchain_type" is sufficient, since "scala" is already in the package name, and it'll make your toolchain type slightly smaller.
Ie,
@io_bazel_rules_scala//scala:scala_toolchain_type
vs
@io_bazel_rules_scala//scala:toolchain_type
This will also be consistent with the go, rust, cc, and jdk toolchain types.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we change to just toolchain_type
as katre suggests?
@katre question: in scala, there are multiple backends: js, native, jvm and different versions that are incompatible (2.10, 2.11, 2.12, ....) Are toolchains a way we build for multiple targets? For instance, I want to build for both jvm and js? So, I guess my question is: could I make a macro that builds the same rule with two different toolchains? Critically, not all scala rules can be built with scalajs because all the transitive dependencies need to be either scalajs or js (you can't depend on a jvm library in scalajs). Similarly for scala-native (it can depend only on pure scala or C libraries). How would you suggest we tackle this issue? |
It depends on how the results would be used. I would probably model the different result types as different platforms: scala_jvm, scala_js, scala_native_x86. Each of these would then have a constraint for the scala compiler backend: In scala/constraints/BUILD.bazel"
In scala/platforms/BUILD.bazel:
And, finally, in scala/toolchains/BUILD.bazel:
Then users can select which toolchain by passing "--platforms=@io_bazel_rules_scala//scala/platforms:scala_jvm" or "...//platforms:scala_js". The rules and toolchain would need logic to tell if a target cannot be built with that toolchain, and to emit an error if the wrong source is given. If jvm vs js vs native output isn't what the user should be selecting with the --platforms, we can work out what the right conceptual model should be. |
This model doesn't allow cross building right?
Being able to build the same graph for scala 2.11 and 2.12 for example
…On Fri, 15 Dec 2017 at 22:29 katre ***@***.***> wrote:
It depends on how the results would be used. I would probably model the
different result types as different platforms: scala_jvm, scala_js,
scala_native_x86. Each of these would then have a constraint for the scala
compiler backend:
In scala/constraints/BUILD.bazel"
constraint_setting(name = "compiler_backend")
constraint_value(name="compiler_jvm", constraint-setting=":compiler_backend")
constraint_value(name="compiler_js", constraint-setting=":compiler_backend")
constraint_value(name="compiler_native", constraint-setting=":compiler_backend")
In scala/platforms/BUILD.bazel:
platform(name="scala_jvm",
constraint_values = ["//scala/constraints:compiler_jvm"])
platform(name="scala_js",
constraint_values = ["//scala/constraints:compiler_js"])
platform(name="scala_native_x86",
constraint_values = [
"//scala/constraints:compiler_native",
***@***.***_tools//platforms:x86_64",
])
And, finally, in scala/toolchains/BUILD.bazel:
scala_toolchain(name = "toolchain_jvm",
other config)
toolchain(name="register_toolchain_jvm",
toolchain_type = "...:toolchain_type",
exec_compatible_with = [],
target_compatible_with = ["//scala/constraints:compiler_jvm"],
toolchain = ":toolchain_jvm")
# Repeat for JS and native
Then users can select which toolchain by passing
***@***.***_bazel_rules_scala//scala/platforms:scala_jvm" or
"...//platforms:scala_js".
The rules and toolchain would need logic to tell if a target cannot be
built with that toolchain, and to emit an error if the wrong source is
given.
If jvm vs js vs native output isn't what the user should be selecting with
the --platforms, we can work out what the right conceptual model should be.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#364 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABUIF_vvjv8rffAB2jTuZeDlE3b3WYRhks5tAtafgaJpZM4Q9pYk>
.
|
Currently, bazel can't support that. It's on our roadmap (you'd add the scala versions as constraints in the platforms and toolchains, and specify multiple target platforms with the "--platforms" flag), but we're not ready to start investigating the work, primarily because it will cause Bazel's internal data structures to double or more in size. |
Sounds good.
Especially since one can use macros to generate duplicate targets
…On Fri, 15 Dec 2017 at 23:28 katre ***@***.***> wrote:
Currently, bazel can't support that. It's on our roadmap (you'd add the
scala versions as constraints in the platforms and toolchains, and specify
multiple target platforms with the "--platforms" flag), but we're not ready
to start investigating the work, primarily because it will cause Bazel's
internal data structures to double or more in size.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#364 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABUIFxgsAjSyvoxjXGTMYhwvmVY5wfOUks5tAuSUgaJpZM4Q9pYk>
.
|
This seems mergeable (as a first step). Is that right? |
@johnynek - sorry for neglecting this thread. Don't merge it yet. I want to update it a bit to follow the suggestions to better naming and then we can merge. If I won't be able to do it in the next few hours I'll surely do it by this coming Tuesday |
Could not understand how to load current repo as external repo within the test
…ain" This reverts commit d7eed19.
I replied on #4372, take a look there. |
close to rerun travis :-( |
open to rerun travis :-( |
Can one of the admins verify this patch? |
test this please (ci bot) |
use cli flag to load the toolchain that should fail the test
- added "default" prefix to make it clear that its default_toolchain - moved the toolchain_type to the top level
Test this please |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great, thanks!
README.md
Outdated
scala_register_toolchains() | ||
``` | ||
[read more here](#scala_toolchain) | ||
# register default scala toolchain |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do we need this section? it looks like the two lines above and it doesn't seem formatted in the "view" mode
@or-shachar Thanks! We've been waiting a long time for this :) |
Thanks to you guys for picking up the work, I'm always happy to help out. |
WIP for #170
Initial implementation of scala_toolchain
BREAKING change - when merged we must register scala toolchain by either:
A) a call to
register_scala_toolchains
must happen, just like inrules_go
, in order to load default toolchain.B) defining custom
scala_toolchain
and registering itOpen questions:
rules_scala
. In ideal case - we want to load the toolchain defined in//test_expect_failure/scalacopts_from_toolchain:failing_scala_toolchain
.Still need to document