Create and publish a MergedManifest for unified builds

[dkurepa]

After we're done with #4198, we'll have a VerticalManifest published for each leg. We'll need to use all of these manifests to produce a MergedManifest, similarly to how we do it currently in individual repos.
We'll need to have some custom logic for this, as multiple legs might produce duplicate assets #4157.

[dkurepa]

Based on the discussions we had offline, we will implement a task that will be used in two scenarios:

• In the join point builds, it will prepare the machine for builds, by downloading necessary assets
• In the final join point it will create a merged manifest and download all assets, preparing for artifact publishing (BlobArtifacts and PackageArtifacts)

Based on these requirements, the task will do the following

• The input will be the name of the "main" vertical manifest, and the list of all vertical manifests
• The task will create a merged manifest by taking all artifacts from the main manifest, and then going through all other manifests and adding artifacts that are not there already
• As it's going through these manifests, it will download pipeline artifacts that corelate to the entries it added to the manifest (it will always download the full main vertical, and then, for example, it takes Microsoft.NETCore.App.Crossgen2.linux-x64 from the Ubuntu2204_x64 vertical`)
• The final result of this task should be a MergedManifest, and build artifacts that correlate to the entries in it. These artifacts will be used for the join point builds, or we'll publish them as BlobArtifacts and PackageArtifacts which are special artifacts that are used by the Maestro publishing pipeline. The MergedManifest will only be used in the final join point, to produce a final MergedManifest. This manifest will be passed to the PushMetadataToBuildAssetRegistry task to create a bar build

[dkurepa]

For downloading pipeline artifacts, we have two options:

• Figure out which artifacts we want to download for each leg, and download them one by one
• Download whole artifact legs, copy what we need, and delete the rest

The main advantage with the first approach is that we'd be downloading less amount of data, we'd be taking just what we need. But this approach also has a fatal flaw, we'd download every file individually so, we'd probably exceed our rate limit, which would cause throttling. Because of this we probably shouldn't use this approach.

The other approach flaw is that we might run out of space, since it requires downloading an artifact zip, and then unzipping it. We'd just save the packages we need, deleting the reset, and could optimize around download order to minimize this risk

[mmitche]

The main advantage with the first approach is that we'd be downloading less amount of data, we'd be taking just what we need. But this approach also has a fatal flaw, we'd download every file individually so, we'd probably exceed our rate limit, which would cause throttling. Because of this we probably shouldn't use this approach.

We don't hit this rate limit when doing our BAR publishing today, so it's quite possible we wouldn't hit it.

[dkurepa]

we're going to pause this work for a bit, and pick it up at a later point

[MilenaHristova]

I'm going to move the changes from dotnet/installer#19369 to sdk but I also want to add some tests for the tasks that @dkurepa implemented (JoinVerticals)

@MichaelSimons I heard there is a limitation on testing the tasks in https://github.com/dotnet/sdk/tree/main/src/SourceBuild/content because of source-build, could you explain why is it a issue? Also is it a good place to test the join verticals functionality in the sdk build as it will be part of the sdk repo now?

[MilenaHristova]

@mmitche in the final pass copying assets to a flat layout is having issues with productVersion.txt files - there is Sdk\9.0.100-rc.1.24414.1\productVersion.txt, WindowsDesktop\9.0.0-rc.1.24413.5\productVersion.txt, aspnetcore\Runtime\9.0.0-rc.1.24412.15\productVersion.txt

But I see there is also Sdk\9.0.100-rc.1.24414.1\sdk-productVersion.txt, WindowsDesktop\9.0.0-rc.1.24413.5\windowsdesktop-productVersion.txt, aspnetcore\Runtime\9.0.0-rc.1.24412.15\aspnetcore-productVersion.txt

If we have those files maybe we can just copy the first productVersion.txt? Or the SDK one?

[MichaelSimons]

@MichaelSimons I heard there is a limitation on testing the tasks in https://github.com/dotnet/sdk/tree/main/src/SourceBuild/content because of source-build, could you explain why is it a issue? Also is it a good place to test the join verticals functionality in the sdk build as it will be part of the sdk repo now?

The limitation is there is a lack of infrastructure for this. The tests do not belong in the smoke tests or scenario tests. We need a new test suite for these types of unit tests - see #4244.

[mmitche]

Oh jeez...I'm not sure why we produce those files. Let me dig a bit and see whether I can figure it out.

[MilenaHristova]

Done with dotnet/sdk#43627