Skip to content

With build -subset help output "macro" subset components #64578

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 1, 2022

Conversation

BruceForstall
Copy link
Contributor

@BruceForstall BruceForstall commented Jan 31, 2022

Subsets like "clr", "mono", "host" are "macro" subset names that imply several other subset names. To try to explain how these relate to all the other names in build -subset help, actually output what they "expand" into.

e.g.,

  Accepted Subset values:
  - Clr
      The full CoreCLR runtime. Equivalent to: clr.native+linuxdac+clr.corelib+clr.tools+clr.nativecorelib+clr.packages+clr.nativeaotlibs
  - Clr.NativePrereqs
      Managed tools that support building the native components of the runtime (such as DacTableGen).
  - Clr.ILTools
      The CoreCLR IL tools (ilasm/ildasm).
  - Clr.Runtime
      The CoreCLR .NET runtime. Includes clr.jit, clr.iltools, clr.hosts.
  - Clr.Native
      All CoreCLR native non-test components, including the runtime, jits, and other native tools. Includes clr.hosts, clr.runtime, clr.jit, clr.alljits, clr.paltests, clr.iltools, clr.nativeaotruntime, clr.spmi.
  - Clr.NativeAotLibs
      The CoreCLR native AOT CoreLib and other low level class libraries.
  - Clr.NativeAotRuntime
      The stripped-down CoreCLR native AOT runtime.
  - Clr.PalTests [only runs on demand]
      The CoreCLR PAL tests.
  - Clr.PalTestList [only runs on demand]
      Generate the list of the CoreCLR PAL tests. When using the command line, use Clr.PalTests instead.
  - Clr.Hosts
      The CoreCLR corerun test host.
  - Clr.Jit
      The JIT for the CoreCLR .NET runtime.
  - Clr.AllJits
      All of the cross-targeting JIT compilers for the CoreCLR .NET runtime.
  - Clr.Spmi
      SuperPMI, a tool for CoreCLR JIT testing.
  - Clr.CoreLib
      The managed System.Private.CoreLib library for CoreCLR.
  - Clr.NativeCoreLib
      Run crossgen on System.Private.CoreLib library for CoreCLR.
  - Clr.Tools
      Managed tools that support CoreCLR development and testing.
  - Clr.Packages
      The projects that produce NuGet packages for the CoreCLR runtime, crossgen, and IL tools.
  - LinuxDac
      The cross-OS Windows->libc-based Linux DAC. Skipped on x86.
  - AlpineDac [only runs on demand]
      The cross-OS Windows->musl-libc-based Linux DAC. Skipped on x86
  - CrossDacPack [only runs on demand]
      Packaging of cross OS DAC. Requires all assets needed to be present at a folder specified by . See 'Microsoft.CrossOsDiag.Private.CoreCLR.proj' for details.
  - Mono
      The Mono runtime and CoreLib. Equivalent to: mono.runtime+mono.corelib+mono.packages
  - Mono.Runtime
      The Mono .NET runtime.
  - Mono.AotCross
      The cross-compiler runtime for Mono AOT.
  - Mono.CoreLib
      The managed System.Private.CoreLib library for Mono.
  - Mono.Packages
      The projects that produce NuGet packages for the Mono runtime.
  - Mono.WasmRuntime
      The WebAssembly runtime.
  - Mono.MsCorDbi
      The implementation of ICorDebug interface.
  - Mono.Workloads [only runs on demand]
      Builds the installers and the insertion metadata for Blazor workloads.
  - Host
      The .NET hosts, packages, hosting libraries, and tests. Equivalent to: host.native+host.tools+host.pkg+host.tests
  - Host.Native
      The .NET hosts.
  - Host.Pkg
      The .NET host packages.
  - Host.Tools
      The .NET hosting libraries.
  - Host.Tests
      The .NET hosting tests.
  - Libs
      The libraries native part, refs and source assemblies, test infra and packages, but NOT the tests (use Libs.Tests to request those explicitly). Equivalent to: libs.native+libs.ref+libs.src+libs.pretest
  - Libs.Native
      The native libraries used in the shared framework.
  - Libs.Ref
      The managed reference libraries.
  - Libs.Src
      The managed implementation libraries.
  - Libs.PreTest
      Test assets which are necessary to run tests.
  - Libs.Packages
      The projects that produce NuGet packages from libraries.
  - Libs.Tests [only runs on demand]
      The test projects. Note that building this doesn't execute tests: you must also pass the '-test' argument.
  - Packs
      Builds the shared framework packs, archives, bundles, installers, and the framework pack tests. Equivalent to: packs.product+packs.tests
  - Packs.Product
      Builds the shared framework packs, archives, bundles, and installers.
  - Packs.Installers
      Builds the shared framework bundles and installers.
  - Packs.Tests
      The framework pack tests.
  - publish [only runs on demand]
      Generate asset manifests and prepare to publish to BAR.
  - RegenerateDownloadTable [only runs on demand]
      Regenerates the nightly build download table

@ghost
Copy link

ghost commented Jan 31, 2022

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

Subsets like "clr", "mono", "host" are "macro" subset names that imply several other subset names. To try to explain how these relate to all the other names in build -subset help, actually output what they "expand" into.

e.g.,

  Accepted Subset values:
  - Clr
      The full CoreCLR runtime: clr.native+linuxdac+clr.corelib+clr.tools+clr.nativecorelib+clr.packages+clr.nativeaotlibs+clr.spmi
  - Clr.NativePrereqs
      Managed tools that support building the native components of the runtime (such as DacTableGen).
  - Clr.ILTools
      The CoreCLR IL tools.
  - Clr.Runtime
      The CoreCLR .NET runtime.
  - Clr.Native
      All CoreCLR native non-test components, including the runtime, jits, and other native tools.
  - Clr.NativeAotLibs
      The CoreCLR native AOT CoreLib and other low level class libraries.
  - Clr.NativeAotRuntime
      The stripped-down CoreCLR native AOT runtime.
  - Clr.PalTests [only runs on demand]
      The CoreCLR PAL tests.
  - Clr.PalTestList [only runs on demand]
      Generate the list of the CoreCLR PAL tests. When using the command line, use Clr.PalTests instead.
  - Clr.Hosts
      The CoreCLR corerun test host.
  - Clr.Jit
      The JIT for the CoreCLR .NET runtime.
  - Clr.AllJits
      All of the cross-targeting JIT compilers for the CoreCLR .NET runtime.
  - Clr.Spmi
      SuperPMI, a tool for CoreCLR JIT testing.
  - Clr.CoreLib
      The managed System.Private.CoreLib library for CoreCLR.
  - Clr.NativeCoreLib
      Run crossgen on System.Private.CoreLib library for CoreCLR.
  - Clr.Tools
      Managed tools that support CoreCLR development and testing.
  - Clr.Packages
      The projects that produce NuGet packages for the CoreCLR runtime, crossgen, and IL tools.
  - LinuxDac
      The cross-OS Windows->libc-based Linux DAC. Skipped on x86.
  - AlpineDac [only runs on demand]
      The cross-OS Windows->musl-libc-based Linux DAC. Skipped on x86
  - CrossDacPack [only runs on demand]
      Packaging of cross OS DAC. Requires all assets needed to be present at a folder specified by . See 'Microsoft.CrossOsDiag.Private.CoreCLR.proj' for details.
  - Mono
      The Mono runtime and CoreLib: mono.runtime+mono.corelib+mono.packages
  - Mono.Runtime
      The Mono .NET runtime.
  - Mono.AotCross
      The cross-compiler runtime for Mono AOT.
  - Mono.CoreLib
      The managed System.Private.CoreLib library for Mono.
  - Mono.Packages
      The projects that produce NuGet packages for the Mono runtime.
  - Mono.WasmRuntime
      The WebAssembly runtime.
  - Mono.MsCorDbi
      The implementation of ICorDebug interface.
  - Mono.Workloads [only runs on demand]
      Builds the installers and the insertion metadata for Blazor workloads.
  - Host
      The .NET hosts, packages, hosting libraries, and tests: host.native+host.tools+host.pkg+host.tests
  - Host.Native
      The .NET hosts.
  - Host.Pkg
      The .NET host packages.
  - Host.Tools
      The .NET hosting libraries.
  - Host.Tests
      The .NET hosting tests.
  - Libs
      The libraries native part, refs and source assemblies, test infra and packages, but NOT the tests (use Libs.Tests to request those explicitly): libs.native+libs.ref+libs.src+libs.pretest
  - Libs.Native
      The native libraries used in the shared framework.
  - Libs.Ref
      The managed reference libraries.
  - Libs.Src
      The managed implementation libraries.
  - Libs.PreTest
      Test assets which are necessary to run tests.
  - Libs.Packages
      The projects that produce NuGet packages from libraries.
  - Libs.Tests [only runs on demand]
      The test projects. Note that building this doesn't execute tests: you must also pass the '-test' argument.
  - Packs
      Builds the shared framework packs, archives, bundles, installers, and the framework pack tests: packs.product+packs.tests
  - Packs.Product
      Builds the shared framework packs, archives, bundles, and installers.
  - Packs.Installers
      Builds the shared framework bundles and installers.
  - Packs.Tests
      The framework pack tests.
  - publish [only runs on demand]
      Generate asset manifests and prepare to publish to BAR.
  - RegenerateDownloadTable [only runs on demand]
      Regenerates the nightly build download table
Author: BruceForstall
Assignees: BruceForstall
Labels:

area-Infrastructure-libraries

Milestone: -

@BruceForstall
Copy link
Contributor Author

Is this actually correct? Are there other "macros" that I missed? Does anyone else think this is helpful?

@jkoritzinsky @MichalStrehovsky @hoyosjs @safern

Copy link
Member

@jkoritzinsky jkoritzinsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As part of this, we should also update any subsets that are not included in these subsets to have the OnDemand="true" metadata.

@@ -101,7 +101,7 @@

<ItemGroup>
<!-- CoreClr -->
<SubsetName Include="Clr" Description="The CoreCLR runtime, LinuxDac, CoreLib (+ native), tools and packages." />
<SubsetName Include="Clr" Description="The full CoreCLR runtime: $(DefaultCoreClrSubsets)" />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MichalStrehovsky @jkotas Does it make sense that build.cmd clr builds nativeaotlibs but not nativeaotruntime?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clr includes clr.native and that's all native code under src/coreclr including clr.nativeaotruntime, clr.iltools, etc.

I just realized I put clr.spmi into the DefaultCoreClrSubsets but that one is also redundant with clr.native. We could remove it; it's a NOP.

@@ -101,7 +101,7 @@

<ItemGroup>
<!-- CoreClr -->
<SubsetName Include="Clr" Description="The CoreCLR runtime, LinuxDac, CoreLib (+ native), tools and packages." />
<SubsetName Include="Clr" Description="The full CoreCLR runtime: $(DefaultCoreClrSubsets)" />
<SubsetName Include="Clr.NativePrereqs" Description="Managed tools that support building the native components of the runtime (such as DacTableGen)." />
<SubsetName Include="Clr.ILTools" Description="The CoreCLR IL tools." />
<SubsetName Include="Clr.Runtime" Description="The CoreCLR .NET runtime." />
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@BruceForstall do you feel it makes sense that the default coreclr set includes SPMI?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's probably not required; we'd just need to make sure all the JIT automation that requires it does actually build it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we'd just need to make sure all the JIT automation that requires it does actually build it.

Although maybe #64035 already did everything that is necessary w.r.t. CI changes required.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dotnet/jit-contrib There's a proposal here to remove "clr.spmi" (the build of the SuperPMI tools) from the default "clr" component build.

(I'm not sure how the "clr.native" component works. It looks like if you use "clr.native" then src\coreclr\runtime.proj calls build-runtime.cmd/sh with no "-component" arguments, so the __CMakeTarget ends up being install, which builds everything? So using "clr.native" will still build everything? But "clr" uses "clr.native", so not sure how we actually "remove" clr.spmi from the "clr" component.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd have to make clr.native in the default be clr.runtime+clr.iltools basically. I was just thinking that most people rarely use SPMI. Not sure libs dev should learn the long incantation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @hoyosjs's question is: Should clr.runtime include the clr.spmi subset? I think having clr build all of the native code is still the correct thing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One could make the same argument about whether the clr should include the Linux DAC. I've never in my life used it and it takes longer to build than spmi. We also build the JIT 6 times because crossgen2/NativeAOT want to be able to crosstarget, but most people do not crosstarget in their day-to-day (crossgen2 devs included).

I think most devs who do daily work in the clr tree already have their own invocations of the build script and don't build all of clr. The clr subset is good for people new to the project that don't know what they want. So they just get everything.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dotnet/runtime-infrastructure Do we see build clr as the product build, or as the inner loop? yml can easily have the longer invocation as needed, but I feel the short one should really be catering to the common case scenario (runtime + corelib) + crossgening maybe even optional. Probably shouldn't be part of this PR, but that could enable an easier workflow for a lot of people

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the model where build clr is the "easy; build everything (related to coreclr)" model, and as you get more sophisticated, you should be able to only build what you care about by using some longer incantation.

Remove `clr.spmi` from default list; it's already there due to `clr.native`.
@BruceForstall
Copy link
Contributor Author

As part of this, we should also update any subsets that are not included in these subsets to have the OnDemand="true" metadata.

I couldn't find any (in CLR) that were not correctly marked (by my understanding).

@BruceForstall
Copy link
Contributor Author

All: I updated the help text, and added documentation text for clr.native and clr.runtime, which include other subsets due to how the build-runtime script (and runtime.proj) are implemented.

This is ready for review.

Copy link
Member

@hoyosjs hoyosjs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from the coreclr perspective. Also, OnDemand seems up to date for the src/coreclr components.

@ghost
Copy link

ghost commented Feb 1, 2022

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Subsets like "clr", "mono", "host" are "macro" subset names that imply several other subset names. To try to explain how these relate to all the other names in build -subset help, actually output what they "expand" into.

e.g.,

  Accepted Subset values:
  - Clr
      The full CoreCLR runtime. Equivalent to: clr.native+linuxdac+clr.corelib+clr.tools+clr.nativecorelib+clr.packages+clr.nativeaotlibs
  - Clr.NativePrereqs
      Managed tools that support building the native components of the runtime (such as DacTableGen).
  - Clr.ILTools
      The CoreCLR IL tools (ilasm/ildasm).
  - Clr.Runtime
      The CoreCLR .NET runtime. Includes clr.jit, clr.iltools, clr.hosts.
  - Clr.Native
      All CoreCLR native non-test components, including the runtime, jits, and other native tools. Includes clr.hosts, clr.runtime, clr.jit, clr.alljits, clr.paltests, clr.iltools, clr.nativeaotruntime, clr.spmi.
  - Clr.NativeAotLibs
      The CoreCLR native AOT CoreLib and other low level class libraries.
  - Clr.NativeAotRuntime
      The stripped-down CoreCLR native AOT runtime.
  - Clr.PalTests [only runs on demand]
      The CoreCLR PAL tests.
  - Clr.PalTestList [only runs on demand]
      Generate the list of the CoreCLR PAL tests. When using the command line, use Clr.PalTests instead.
  - Clr.Hosts
      The CoreCLR corerun test host.
  - Clr.Jit
      The JIT for the CoreCLR .NET runtime.
  - Clr.AllJits
      All of the cross-targeting JIT compilers for the CoreCLR .NET runtime.
  - Clr.Spmi
      SuperPMI, a tool for CoreCLR JIT testing.
  - Clr.CoreLib
      The managed System.Private.CoreLib library for CoreCLR.
  - Clr.NativeCoreLib
      Run crossgen on System.Private.CoreLib library for CoreCLR.
  - Clr.Tools
      Managed tools that support CoreCLR development and testing.
  - Clr.Packages
      The projects that produce NuGet packages for the CoreCLR runtime, crossgen, and IL tools.
  - LinuxDac
      The cross-OS Windows->libc-based Linux DAC. Skipped on x86.
  - AlpineDac [only runs on demand]
      The cross-OS Windows->musl-libc-based Linux DAC. Skipped on x86
  - CrossDacPack [only runs on demand]
      Packaging of cross OS DAC. Requires all assets needed to be present at a folder specified by . See 'Microsoft.CrossOsDiag.Private.CoreCLR.proj' for details.
  - Mono
      The Mono runtime and CoreLib. Equivalent to: mono.runtime+mono.corelib+mono.packages
  - Mono.Runtime
      The Mono .NET runtime.
  - Mono.AotCross
      The cross-compiler runtime for Mono AOT.
  - Mono.CoreLib
      The managed System.Private.CoreLib library for Mono.
  - Mono.Packages
      The projects that produce NuGet packages for the Mono runtime.
  - Mono.WasmRuntime
      The WebAssembly runtime.
  - Mono.MsCorDbi
      The implementation of ICorDebug interface.
  - Mono.Workloads [only runs on demand]
      Builds the installers and the insertion metadata for Blazor workloads.
  - Host
      The .NET hosts, packages, hosting libraries, and tests. Equivalent to: host.native+host.tools+host.pkg+host.tests
  - Host.Native
      The .NET hosts.
  - Host.Pkg
      The .NET host packages.
  - Host.Tools
      The .NET hosting libraries.
  - Host.Tests
      The .NET hosting tests.
  - Libs
      The libraries native part, refs and source assemblies, test infra and packages, but NOT the tests (use Libs.Tests to request those explicitly). Equivalent to: libs.native+libs.ref+libs.src+libs.pretest
  - Libs.Native
      The native libraries used in the shared framework.
  - Libs.Ref
      The managed reference libraries.
  - Libs.Src
      The managed implementation libraries.
  - Libs.PreTest
      Test assets which are necessary to run tests.
  - Libs.Packages
      The projects that produce NuGet packages from libraries.
  - Libs.Tests [only runs on demand]
      The test projects. Note that building this doesn't execute tests: you must also pass the '-test' argument.
  - Packs
      Builds the shared framework packs, archives, bundles, installers, and the framework pack tests. Equivalent to: packs.product+packs.tests
  - Packs.Product
      Builds the shared framework packs, archives, bundles, and installers.
  - Packs.Installers
      Builds the shared framework bundles and installers.
  - Packs.Tests
      The framework pack tests.
  - publish [only runs on demand]
      Generate asset manifests and prepare to publish to BAR.
  - RegenerateDownloadTable [only runs on demand]
      Regenerates the nightly build download table
Author: BruceForstall
Assignees: BruceForstall
Labels:

area-Infrastructure

Milestone: -

@BruceForstall BruceForstall merged commit 5e38aec into dotnet:main Feb 1, 2022
@BruceForstall BruceForstall deleted the OutputSubsetComponents branch February 1, 2022 23:14
@ghost ghost locked as resolved and limited conversation to collaborators Mar 4, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants