Proposal: Address Project Reunion goals by addressing Visual Studio restrictions

[Scottj1s]

Proposal: Address Project Reunion goals by addressing Visual Studio restrictions

Summary

Many of the barriers to code sharing and interop that Project Reunion seeks to remove are either active restrictions, or passive limitations, of Visual Studio. These have been surfaced with the release of new WinRT projections like C++/WinRT and C#/WinRT.

Rationale

Project Reunion is likely to highlight these Visual Studio restrictions even more, and so should address their elimination. These restrictions fall into build time and run time categories, detailed below. This proposal is something of a grab bag of known issues, and seeks to prioritize their resolution so that the end to end developer experience is more streamlined.

Build Time

C++ PackageReferences

C++ projects (*.vcxproj files) do not support Nuget PackageReferences, but only packages.config references, which have a number of limitations:

• They must target a specific package version, even with allowedVersions
• They don't participate with the project system, so can't import other artifacts or use replaceable parameters
• They don't support 'msbuild /t:restore', so a separate 'nuget restore' is necessary in build pipelines, etc

There is a longstanding open request to add this support:
Use PackageReference in vcxproj
And a related longstanding open PR to implement it:
add C++ PackageReference support

Desktop <--> UWP Project References

Project references cannot be added in Visual Studio between Desktop and UWP (Universal Windows) projects. Attempts to do so result in an error: A reference cannot be added because the two projects target different platforms.. This can be worked around with a project file edit, or introduction of a directory.build.* file, to manually add the project reference. In other words, this is strictly a tooling restriction, which prevents interop unless developers are aware of the workarounds.

Runtime

Unified CRT

Having enabled Desktop<-->UWP project references, the developer must then address the runtime conflict of C Runtime libraries used by both modules. A Desktop module imports C:\Windows\System32\vcruntime140.dll and friends, while a UWP module imports C:\Program Files\WindowsApps\Microsoft.VCLibs.140*\vcruntime140_app.dll and friends. This impedance mismatch can be addressed by using the VCRT Forwarders package. But this technique is not very discoverable (it's not obvious that CRT dynalink errors can be addressed with a special adapter Nuget package).

A better solution is the inclusion of both CRT flavors in a unified VCLibs.UWPDesktop framework package, usable by both packaged UWP and centennial modules. This would obsolete the need for the VCRT Forwarders package, which is an explicit user action that adds build and deploy complexity, and increases install image size. Visual Studio could ensure that all binaries are "chameleon linked" with the CRT, using the Desktop import entries. Selection between Desktop CRT and unified VCLibs framework package could then be deferred until runtime, enabling broad reusability of modules. Visual Studio could also automatically select the unified VCLibs framework package for debugging and packaging behavior of UWP projects.

Native WinRT Activation

Having addressed the CRT linkage issues, there are often WinRT activation issues. Historically, activation of user-defined WinRT components was restricted to packaged apps. With Windows 1903 (19H1), support was added for unpackaged app activation, based on a fusion manifest: Enhancing Non-packaged Desktop Apps using Windows Runtime Components. This requires additional user action that is not discoverable and is tedious - the manifest requires an entry for every activatable class. Visual Studio could include project templates and/or build-time customization to automatically generate and populate this manifest.

There is an effort to address the Windows 1903 requirement of the Reg-Free Activation feature above, using a Detours-based library: Undocked RegFree Winrt Activation. Ideally, this library could be delivered as a Nuget package, along with build-time customizations mentioned above, and included in Visual Studio project templates.

Managed WinRT Activation

The above discussion addresses activation of native components, from either native or managed code. There is some support in Visual Studio for the opposite - activating managed components from native code. But this is subject to a number of limitations.

For example, a packaged UWP app can include a project reference to a managed UWP component and Visual Studio will generate the necessary uwpshims.exe appx manifest entry for hosting the managed component. However, there is a bug in this support that requires targeting Windows SDK 15063 (RS2) or older: Enable C++ Hybrid apps to target .Net native 1.7 and 2.2 without workarounds.

For activating a managed component from a native Desktop (non-packaged) app, there is currently no support in Visual Studio. An approach similar to Reg-Free Activation above (using a fusion manifest, in the absence of any other app manifest), could be used to implement shim support similar to that for packaged apps above.

[stevenbrix]

So a) thank you for filing this issue Scott! This is a perfect summary of many of the tooling pain points our customers are facing. The good news is that I believe we are starting to address many of these. More details on each below.

C++ Package References

@davkean said in a meeting that dotnet/project-system#2491 would be in VS 16.8 and bring with it PackageReference support for C++. Is this still the plan of record? The lack of updates on the issue worries me a bit that this may not be coming.

Desktop <--> UWP Project References

This is being addressed right now with WinUI3. We've introduced a DesktopCompatible property that allows the ProjectReference to work. If you're using VS16.7, you can set this property in your .vcxproj.

Native WinRT Activation

We have a task to make this work for WinUI3 apps: https://microsoft.visualstudio.com/OS/_workitems/edit/25174345/

We'll be a shipping a custom MSBuild task in the Microsoft.WinUI nuget that will do this. If there is a better place that this code should live/ship so that it isn't only WinUI apps that can use it, then we're open to that.

Unified CRT

I love this, I don't have much to add other than needing to ensure that we support the self-contained xcopy deployment scenario that will place the appropriate vcruntime binaries next to the .exe

[stevenbrix]

I forgot a b...

and b) Hooray for Project Reunion! 🙌 😄

[mikebattista]

Unified CRT
Having enabled Desktop<-->UWP project references, the developer must then address the runtime conflict of C Runtime libraries used by both modules. A Desktop module imports C:\Windows\System32\vcruntime140.dll and friends, while a UWP module imports C:\Program Files\WindowsApps\Microsoft.VCLibs.140*\vcruntime140_app.dll and friends. This impedance mismatch can be addressed by using the VCRT Forwarders package. But this technique is not very discoverable (it's not obvious that CRT dynalink errors can be addressed with a special adapter Nuget package).

A better solution is the inclusion of both CRT flavors in a unified VCLibs.UWPDesktop framework package, usable by both packaged UWP and centennial modules. This would obsolete the need for the VCRT Forwarders package, which is an explicit user action that adds build and deploy complexity, and increases install image size. Visual Studio could ensure that all binaries are "chameleon linked" with the CRT, using the Desktop import entries. Selection between Desktop CRT and unified VCLibs framework package could then be deferred until runtime, enabling broad reusability of modules. Visual Studio could also automatically select the unified VCLibs framework package for debugging and packaging behavior of UWP projects.

@stwish-msft is working on a single VC libs package that works everywhere.

[Scottj1s]

@stevenbrix - DesktopCompatible sounds promising but it should not be specific to WinUI projects, and so should be implemented in VS project system support. PackageReference for C++ still has some resistance to overcome, so we'll see if it lands.

@mikebattista - Thanks for making that statement public. I wasn't sure if it had been.

[stevenbrix]

@stevenbrix - DesktopCompatible sounds promising but it should not be specific to WinUI projects, and so should be implemented in VS project system support.

@Scottj1s it is in the VS project system ☺️. It isn't WinUI specific.

[kennykerr]

Along the same lines, as limitations persist in Visual Studio, developers have increasingly gone elsewhere for their needs, building Windows apps with languages, toolchains, and IDEs that have nothing to do with traditional Visual Studio and msbuild. Some languages like C++ don't have a standard build system, but without good support from Visual Studio, developers are increasingly using cmake. Other languages like Rust include standard build support via cargo. In such cases, nuget and msbuild are completely foreign. Components written for Project Reunion should minimize the friction to make Windows approachable to more developers. A concession can be made to use nuget as the packaging format, while limiting the use of msbuild such that projects written in C++ or Rust can simply extract the component's winmd and DLLs without any further knowledge of the component's implementation and dependencies.

[davkean]

Having been around for Win8/UWP on the Visual Studio sides of things, these "restrictions" around referencing are more better described as Windows business rules (at the time) being enforced by Visual Studio.

There is also a type unification problem, in that .NET Framework projects do not get the WindowsRuntime bridge assemblies between C# <-> WinRT. Allowing the reference might appear to succeed, but without these three assemblies, various things won't work:

System.Runtime.InteropServices.WindowsRuntime.dll
System.Runtime.WindowsRuntime.dll
System.Runtime.WindowsRuntime.UI.Xaml.dll

Has this been addressed, or are there plans to address this?

@stevenbrix: Looking at code, it looks like DesktopCompatible is only respected by C++ and not C#. Is that intentional?

[jonwis]

NuGet vs vcxpkg vs cmake

Seems reasonable for us to have a "Using Project Reunion" wiki page here or sample directory in the repo showing how to integrate the built artifacts with non-Visual Studio / msbuild environments. The Project Reunion team will initially focus on Visual Studio to get going, and we welcome pulls from the community to help us learn about the other project systems.

I think @kennykerr was signing himself up for cmake/Rust integration? :)

[kennykerr]

The point is that we don't want to put the burden on component authors to have to support N different build systems. If we can follow certain simple conventions we can support a large variety of build systems and languages simply by virtue of not having packaging requirements that lock developers into using Visual Studio.

[sylveon]

I don't like the "fix" for the CRT. It only cares about packaged scenarios and unpackaged scenarios will still have to ship the forwarders. Doesn't fix the root issue either.

It is possible to force linking UWP project to the desktop CRT using a property that's undocumented in VS and I've been successfully using it for a while in an unpackaged XAML Island scenario with custom XAML controls (since for now the XAML compiler and tools only light up in UWP projects). No need to ship any forwarder, just the DLL containing the compiled XAML code.

In my opinion, we should fix the root issue of XAML tooling only available in UWP projects (support building XAML within a desktop project), and ship WinUI builds linked against the desktop CRT for use by both packaged and unpackaged Win32 scenarios (since those scenarios already depend on that CRT anyways). That would also remove the need for the DesktopCompatible property for many use cases (although it would still be useful for custom controls which support both Win32 and UWP). Also why did nobody tell me this I could use that property!

PackageReference support would be great. Its absence has been a major pain point to adopting NuGet for C++, and it's obvious that C++ is an afterthought of the NuGet team. I pick vcpkg over NuGet whenever possible for libraries (the only lib I use NuGet for currently is C++/WinRT, by lack of choice) because since C++ is the only actually supported language, the selection of libraries is much bigger, and it handles C++ specific scenarios that NuGet just doesn't (like customizing the library build or building from head). It doesn't handle versionning though which can be a bit annoying when your CI fails because an update was published to a library you depend on which broke the API.

There are also other glaring flaws in C++ support, like the XAML compiler often emitting code that outright doesn't compile when the same XAML in C# does: see microsoft/microsoft-ui-xaml#2721 and microsoft/microsoft-ui-xaml#2429. These should be fixed before the stable release of WinUI 3 because they're pretty basic scenarios that developers who adopt it will run into quickly (hell, I'm writing very simple GUIs in XAML Islands for a simple program and I ran into both without even trying)

These obvious issues make it feel like C++ is a second class citizen of the XAML compiler.

[sylveon]

As for the WinRT activation thing, how would auto generating the manifest as part of a build task work if the user already has a custom app manifest with other things (like Common Controls 6, per monitor v2 DPI, long path awareness, the supportedOS declaration, etc...).

Using Detours feels more like a hack than a proper solution.

[jonwis]

mt.exe knows how to merge multiple "snippets" into a final manifest during build.

[Scottj1s]

Another way to address manifest complexities is to make them optional. C++/WinRT, C#/WinRT and Rust/WinRT all provide reg-free AND manifest-free activation, by simulating what RoGetMetaDataFile does (described under WinRT type activation for C#/WinRT). System reg-free WinRT activation added in 19H1 should do likewise.