Create workload insertion process

[MiYanni]

Related: dotnet/sdk#41607

Summary

The primary goals of this PR are to:

• Update the official.yml build to include a process for:
  • Downloading the workloads associated with the workload set created by this branch (via Arcade via Version.Details.xml/Versions.props)
  • Create the VS drops of the acquired workloads (published to https://vsdrop.microsoft.com)
  • Create the appropriate VS insertion PRs to the desired VS branches (both against the 'current' VS branch and 'prior' branches)
• Update the public.yml to actually build and be used in PR builds to this repo

If I've done my job well, the result should be a lot of dense YAML pipeline logic and PowerShell scripts. There is a significant amount of 'magic' involved in making this function. I'll walk through the details below.

The Process

To preface, I attempted to reuse as much existing circuitry as possible. This means Azure Pipeline tasks that already exist and Arcade build 'enchantments' (🧙). In doing so, there were a lot of hurdles and I used PowerShell to try to do the least amount of work possible to overcome those hurdles. Let's review the steps that the official.yml build goes through.

The Steps

The official.yml is configured to only run a single Build stage. I've created a template called workload-build.yml that does the entire build process.

Why is workload-build.yml a separate template?

• Azure Pipelines does not allow you to hide pipeline parameters. In workload-build.yml, there is a static set of values I needed to be defined as a parameter (not a variable) as those values are required at compile time (variables are not evaluated at compile time). Therefore, the only way to have static, 'hidden' parameter is to encase all the logic in a template, which is why workload-build.yml exists. The parameter in question is workloadDropTypes declared at the top of workload-build.yml.

workload-build.yml

The jobs define in this template are within the jobs.yml context from Arcade. This allows the creation of the Publish Assets job automatically, which pushes the build assets to DARC.

When doing an insertion, we first run the download-workloads.ps1 script.

What does download-workloads.ps1 do?

• This script downloads the workload NuGet packages via DARC. It requires a GitHub PAT and an AzDO PAT to run the DARC command. This script can be ran locally without those explicit PATs as outlined in the README.md. This script also configured to take a workloadListJson, which specifies which workloads to acquire. It does this via a regex sent to the DARC command. The regex also accounts for components versus pre.components, which is designated when running the pipeline using the usePreComponentsForVSInsertion pipeline parameter.

Next, we the "regular build" via the build.ps1 provided by Arcade. This will build the workload-versions.sln. I will not go over the process of the building the workload set JSON via the Microsoft.NET.Workloads.csproj, as that process has not changed in this PR. The new projects added are Microsoft.NET.Workloads.Vsman.csproj and Microsoft.NET.Workloads.Vsman.vsmanproj.

Why are there 2 projects, a .csproj and a .vsmanproj?

• The .csproj only contains one target, BuildVsmanProj. It only runs this target. The .csproj is a bootstrapper to build the .vsmanproj via the create-workload-drops.ps1 script. Before running that script, the BuildVsmanProj target does the local-build version of using the download-workloads.ps1 script (when necessary). After that, it runs create-workload-drops.ps1 which ends up building the .vsmanproj multiple times, once per workload drop acquired from the download-workloads.ps1 script.

What does create-workload-drops.ps1 do?

• This is the main script for creating the workload insertion metadata. It:
  1. Extracts the workload drop zips.
  2. On a loop per drop, uses a regex to parse metadata about the drop out of the drop name.
  3. Computes a unique hash for the files in the drop.
  4. Reads the version information for the drop out of the .metadata file.
  5. Builds the Microsoft.NET.Workloads.Vsman.vsmanproj for the drop, which creates the vsman file for that drop.
  6. Sets the Azure Pipelines variables for the drop metadata, required for subsequent processes in the official build pipeline.
  7. Updates the strings that represent the component list for doing the VS insertion later in the official build pipeline.
  8. After the loop completes, cleans up intermediate build files from the workload drop folders.
  9. Sets the Azure Pipeline variables for the component lists, one for the primary (components & packs) and one for the secondary (packs only).
• Note: Per loop iteration in the script, pipeline variables are dynamically created based on the workload drop's shortname, drop type, and purpose. For example, for the emsdk components drop, variables named emsdk_components_name, emsdk_components_dir, and emsdk_components_url would be created.

After the regular build if the publishToFeed pipeline parameter is set, the pipeline will publish the workload set to the designated AzDO feed via the feedForPublishing pipeline parameter. Currently, this is only related to the workload set package and does nothing with the workload drops.

Now comes the fun part (😈). When the pipeline is doing an insertion, it will utilize compile-time foreach loops over the parameters workloadDropNames and workloadDropTypes. The types-loop is nested inside of the names-loop. This means for each workload drop, it will run all the following tasks for each workload drop type. For reference, the workload drop types are components, packs, and precomponents. This list of types must be statically defined as a parameter to be used with these compile-time foreach loops.

Within this looping mechanism, there are two tasks. The first task is a small, embedded PowerShell script that simply checks to see if the drop exists for the combination of drop name and drop type. This has to be checked as the drop type components and precomponents won't be downloaded at the same time (they are mutually exclusive). Also, there could be a workload designated in the Version.Details that doesn't have a published workload drop yet, so it wouldn't have been downloaded. This script just validates that the drop folder exists on disk.

The second task in the loop is the 1ES.MicroBuildVstsDrop task. This task uploads the workload drop to the VS drop blob storage for use in creating a VS insertion PR. This task only allows you to upload one drop at a time, which is why this task needs to be looped over in the pipeline. This only runs if the previous PowerShell script set the pipeline variable PublishWorkloadDrop to true.

Outside of those nested loops, the last part of workload-build.yml has 2 loops, one per each set of insertion branches designated via the pipeline parameters primaryVsInsertionBranches and secondaryVsInsertionBranches respectively. These are the branches in the VS repo that the primary (component & packs) and secondary (packs only) will be used for VS insertion PRs. In each loop, the workload-insertion.yml template is called.

workload-insertion.yml

This template has two tasks. The first is a small, embedded PowerShell script that sets the InsertionTopicBranch pipeline variable. This variable is calculated based on the value passed in as the topicBranch parameter to this template, which is passed in as the vsTopicBranch pipeline parameter. By default, this value is |default| which resolves to a month-based branch name. Someone could also supply |temp| which creates a branch name using the build number and run attempt for the pipeline (so it always changes). Lastly, if someone supplies any other value, that value is used as the topic branch name.

The second task is MicroBuildInsertVsPayload which is used to create the VS insertion PR. It required creating our own service connection named MicroBuildInsertionVS in the DncEng org to have the proper permissions for this task. This task primarily relies on the component JSON values (supplied via componentJsonValues) to create the VS insertion PR changes. Other than that, every other parameter to the task is fairly self explanatory.

[Forgind]

I didn't approve because I don't feel comfortable really reviewing ps1 files, but the parts I looked at looked good 🙂

[Forgind]

I tried to review this as thoroughly as I could. Hopefully my questions at least make sense 🙂

[MiYanni]

Note: Tested logic

[MiYanni]

Note: Tested logic for VS Drop access

[nagilson]

I've left some questions and thoughts below, mostly nits. Fantastic work and thank you for the descriptive PR information, I can tell this took a lot of work. I am not an expert in this space but hopefully some of this is helpful.