Crossplat MSBuild and path representation

[cdmihai]

This is a part of #781 but focused on path separators and how path values flow through msbuild.

Path values get introduced in MSBuild's data flow from:

• user input (command line properties, properties, items, metadata, task arguments, imports, using tasks, conditions)
• glob expansions (discussed in Crossplat MSBuild and glob expansion #1026 )
• task outputs
• well known item metadata pertaining to paths (FullPath, RootDir, etc)

Paths values are consumed in:

• Conditions (path comparisons, etc)
• Tasks
• Exec task

Path consumers can access paths via:

• direct property / item reference (@(Compile), $(MyFile))
• item file metadata (FullPath, Identity, etc)

Observations:

• There is no distinction between strings and paths
• Path value consumers can interpret the values either as plain strings (and MSBuild should not massage the value thinking it's a path), or as actual paths and thus do IO operations on them (when MSBuild is OK to normalize paths, etc)

Problems:

• crossplatform MSBuild may break if path producers do not agree with path consumers. For example if producers and consumers use different slashes, trailing slashes, absolute vs relative paths, etc

Potential solutions:

• msbuild could normalize all or some path producers in regards to separators to:
  • only use forward or backward slashes
  • use the file system specific path separator

[bleroy]

use the file system specific path separator

That can't work as your typical .NET Core OSS project should be buildable on any system.

What's the objection against making them interchangeable? I get that it could break some existing tasks, but do we know exactly how bad that would be?

[borgdylan]

Currently Mono's xbuild normalizes paths to work on the current platform. Will msbuild do the same or will there be an agreed standard for the path separator in the xml files?

[jonpryor]

My two cents: MSBuild should follow idealized xbuild behavior, by which I mean:

1. Strings and paths are interchangeable. There is no difference.

2. Whenever strings are compared, \ and / compare as the same value, and thus "Foo\Bar" == "Foo/Bar" is True.

3. When reading from/writing to a file, MSBuild will replace \ and / with the correct OS-native directory separator character.

   <ItemGroup>
     <Foo Include="Something\File1.txt" />
     <Foo Include="Something\File2.txt" />
   </ItemGroup>
   <WriteLinesToFile File="Some\Path.txt" Lines="@(Foo)" />
   

   On Unix, the <WriteLinesToFile/> task should write to the file Some/Path.txt, and the Lines property should likewise contain Something/File1.txt and Something/File2.txt.

4. In the (hopefully rare) instances in which a \ is actually required, e.g. to use the <Exec/> with the grep(1) Unix command utility, which uses \ as an escape character, \ should be escaped and instead entered as %5c.

Rationale

Most existing MSBuild projects use \ within filesystem paths, e.g.:

  <Import Project="$(MSBuildExtensionsPath)\Xamarin\Android\Xamarin.Android.CSharp.targets" />

Thus, on Unix platforms, \ must be treated as / most of the time, or the above import won't work. That's Bad™.

(2) is the natural fallout of this: if most project files will be written originally on Windows, if they have any use of Condition which may compare a filesystem path against anything else, then \ and / need to be normalized so that they're treated the same. For example:

<Import
  Condition="'$(Language)' != 'F#' And Exists('$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion)\FSharp\Microsoft.FSharp.Targets')"
  Project="$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion)\FSharp\Microsoft.FSharp.Targets" />

We could argue that only Exists() within a Conditional needs escaping, but I believe that the mental model is easier to reason about if we say that all strings are treated the same, not just "substrings" within certain contexts.

(3) Shows why this needs to be done consistently everywhere. Filesystem paths can be stored in anything, e.g. %(Reference.HintPath), and that "anything" can in turn be passed on to other tasks which will read or write those values. Since any string may potentially contain a path value, all strings need to be similarly munged so that \ or / will contain the OS-native directory separator character.

(4) is the escape hatch: sometimes you need an actual \ or an actual /, for any number of reasons. In such situations, we should require using the existing %xx literal syntax to enter the desired character.

---

Note that I said that this is the idealized xbuild behavior. I have observed scenarios where I would write Item Metadata which contains a \, then invoke the <Copy/> task, the \ would be used as part of the copy output. This...is maddening.

<?xml version="1.0" encoding="utf-8"?>
<!-- b.targets -->
<Project DefaultTargets="Default" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <Dir>Foo</Dir>
  </PropertyGroup>
  <ItemGroup>
    <F Include="b.targets">
      <MD>Foo\Example</MD>
    </F>
  </ItemGroup>
  <Target Name="Default">
    <Message Text="foo\bar" />
    <MakeDir Directories="$(Dir)" />
    <Copy SourceFiles="@(F)" DestinationFiles="%(F.MD)" />
  </Target>
</Project>

When running on xbuild:

$ xbuild b.targest
...
    Target Default:
        foo/bar
        Copying file from '/Users/jon/tmp/b.targets' to '/Users/jon/tmp/Foo\Example'

Just...wat.

MSBuild actually behaves reasonably here.

[mfilippov]

My two cents about xBuild: xBuild just replace char '\' to Path.DirectorySeparatorChar in all sub items: https://github.com/mono/mono/blob/master/mcs/class/Microsoft.Build.Engine/Microsoft.Build.BuildEngine/Expression.cs#L103

[jonpryor]

Addendum/elaboration: the original comment wrote:

Paths values are consumed in:

• Exec task

A related concern is that path values aren't just "consumed" in the <Exec/> task. They are also generated in the <Exec/> task. Consider the xamarin-android repo, which contains <Exec/>s such as:

<Exec Command="%(_LlvmRuntime.ConfigureEnvironment) $(LlvmSourceFullPath)\configure %(_LlvmRuntime.ConfigureFlags)" />
<Exec Command="&quot;%(_MonoRuntime.Strip)&quot; %(_MonoRuntime.StripFlags) &quot;$(OutputPath)\lib\xbuild\Xamarin\Android\lib\%(_MonoRuntime.Identity)\%(_MonoRuntime.OutputRuntimeFilename).%(_MonoRuntime.NativeLibraryExtens
ion)&quot;" />
<Exec Command="$(ManagedRuntime) ..\..\bin\Build$(Configuration)\jnienv-gen.exe -o Android.Runtime\JNIEnv.g.cs --use-java-interop" />
<Exec Command="&quot;$(AndroidNdkDirectory)\ndk-build&quot;" /
<Exec Command="$(AndroidToolchainDirectory)\ndk\ndk-build $(_NdkBuildArgs) NDK_LIBS_OUT=./libs/Debug NDK_OUT=./obj/Debug V=1" />
<Exec Command="&quot;$(JavaSdkDirectory)\bin\javac&quot; -source 1.5 -target 1.6 -d &quot;Jars\classes&quot; @(InputJarSource -&gt; '%(Identity)', ' ')" />
...

My (incorrect?) understanding is that, at present, MSBuild/master will replace \ with / when accessed from Item metadata and properties. This is insufficient. Many of the above commands should work in a cross-platform manner, but if \ replacement only happens for properties and item groups, none of the above will be "fixed" to execute properly on Unix.

All of those execute on Unix, today, using xbuild and the msbuild included with Mono 4.8.

[jeremyVignelles]

Any update on this? I'm in favor of treating \ equal to /, for example when using Include="%2A%2A" with Exclude=a/b/c at the same time, should I use forward or backslashes in exclude?

[livarcocc]

We have chosen to keep the current behavior. At this point, changing this would be a big breaking change for what does not look like a fantastic improvement.