Build places dependent project outputs wrong folder

[analogrelay]

Right now, when you have project A depending on project B, we compile project B directly to project A's output directory. For example, consider the following projects:

A/project.json:

{
    "dependencies": { "B": { "target": "project" } },
    "frameworks": {
        "dnxcore50": {}
    }
}

B/project.json:

{
    "frameworks": {
        "dotnet5.4": {}
    }
}

Running dotnet build on A results in the following files under the A directory:

.
./bin
./bin/Debug
./bin/Debug/dnxcore50
./bin/Debug/dotnet5.4
./bin/Debug/dotnet5.4/B.dll
./bin/Debug/dotnet5.4/B.pdb
./obj
./obj/Debug
./obj/Debug/dnxcore50
./obj/Debug/dnxcore50/dotnet-compile-csc.rsp
./obj/Debug/dnxcore50/dotnet-compile.A.rsp
./obj/Debug/dnxcore50/dotnet-compile.assemblyinfo.cs
./obj/Debug/dnxcore50/dotnet-compile.B.rsp
./Program.cs
./project.json
./project.lock.json

Notice that B was built into the bin/Debug/dotnet5.4 folder (the TFM of B) rather than into the bin/Debug/dnxcore50 folder (the TFM of A, the project being built). As a result, A's compilation then fails because it is expected to find the binary in the dnxcore50 folder.

An example of this is here: https://github.com/anurse/Scratch/tree/master/cli987. Clone the repo and go to that cli987 folder, run dotnet restore, then cd A, then dotnet build. It should succeed. However, it will actually fail with: error CS0006: Metadata file '...path.../cli987/A/bin/Debug/dnxcore50/B.dll' could not be found

/cc @livarcocc @piotrpMSFT @davidfowl

[cdmihai]

[Clarification by @anurse: This comment was originally written when this bug was describing a proposed solution of having dotnet-build compile each project into their own individual project directories, use those paths in the /r: switches passed to compilers, and then perform the copying in a final pass]

This creates a bit of a problem for referencing the project dependencies to the compiler.

Let's say you have three projects, A -> B -> C, and you run dotnet build A --out binA. This means C has to be built first, then B, then A.

C would build into its own private binC directory, and copy its output to binA.
B intends to build into its own private binB directory, but fails because it does not find the dll of project C, because C copied its output to binA

On the other hand, for incrementality this is great news because each output directory has only files relevant to that project. It also enables a potential concurrent build by eliminating racing in the same directory (somewhat)

[analogrelay]

FYI, I've updated the title and description to be more of a problem statement rather than a potential solution, to clarify the desired outcome.

[analogrelay]

@cdmihai I don't actually think that would be a problem with my proposed solution, since the output of compiling project P would always be located in P/bin/Debug/.... My proposal was to have each project compiled to their own directories and then have dotnet build copy them into the final output directory. That wasn't clear in my original post though, and it's probably better to frame this as a discussion of the underlying problem rather than jumping to a solution :)

[victorhurdugaci]

Marking as blocking-partner because we cannot compile our code with the latest CLI.

[livarcocc]

Even if we do copy the binaries to the project individual folder, how do we understand the relation from say dnx451 referencing net451. We need to understand that one is usable by the other and from there we could know from where tfm folder to pick up the dependency from, even if we still keep things compiled wherever, be it the first project folder, or the output folder passed in the command line.

[livarcocc]

It should just follow the lock file, to me, wether it will find files in a single output (appended with config/tfm), or each individual project folder (with config/tfm), I don't really have a strong opinion about. But I do believe that the mapping from one tfm to another should be happening magically as far as compile is concerned. It will just follow the references.

This way, if its target tfm is dnx451 and it depends on a project with net451, this mapping should be in the lock file and compile should reference that dependency with tfm net451. This way, we would still be building each project on their own tfm folders but it would still work.

[davidfowl]

That's how the lock file works already. We just need to produce output paths to the correct place. The reason we eventually copy those p2p refs is to run them. In fact, we need to drop them in a RID specific folder for non class libraries

[muratg]

@livarcocc @piotrpMSFT Hey folks, this is blocking our builds currently. Do you guys have an ETA for fixing it?

cc: @danroth27 @Eilon

[cdmihai]

I also think this inconsistency of target frameworks (project A built for framework X depends on project B built for framework Y) should be caught by Nuget restore, which should output an error like incompatible frameworks detected between dependencies.

If this should not be an error, then there needs to be some logic somewhere with a compatibility table specifying how frameworks are compatible between themselves.
In this case, if TFM X is compatible with TFM Y, Compile needs to be told how to reference the dll of the built project dependencies (how does A find B's output?).

Could the lock file contain this information? If a dependency is actually a project dependency, have the lock file point to its output directory dlls.
Otherwise, there could be a nuget dev cache where project type dependencies get nugetized (project A depends on B, so B gets built, nugetized, moved to the dev cache, where it will be found by A, from A's lock file)

[davidfowl]

I'll take a look at this since it's blocking us.

[victorhurdugaci]

@davidfowl any updates on this one?

[davidfowl]

@victorhurdugaci I'm close.