[release/2.1] Detect package cache blocking source-built package uptake

[dagood]

Detect when a source-built package exists, but a different package exists in the package cache directory. NuGet restore short-circuits when the package id/version specified is already in the cache, so the source-built one won't be used by downstream repos.

---

This is for #606.

It actually spots an active problem in this branch (release/2.1):

Watch out! Source-built nupkg is not byte-for-byte identical to nupkg in cache: Newtonsoft.Json 9.0.1

I checked to make sure contents of the package cache are really from online. Related (although I don't think we realized the extent of the issue): #593.

[dseefeld]

I don't get what's happening with CustomUsageCheckPackageDir here.

[dagood]

Ah, yeah, this is kind of out of nowhere here. I'll add a comment. The idea is it's used by FindSourceBuiltUsages to do a "custom usage check".

[dseefeld]

I see that, but why is it included in PrebuiltPackages?

[dagood]

Heh, good point... I was being lazy and using the existing target without much modification just out of convenience. I'll reorganize this to make sense.

I'll also try to make it run immediately after it detects the problem and focus the report the specific problematic package(s). If you just do what the message suggests and run this after the build completes, there's too much junk to practically sift through.

[dagood]

Now it outputs this in this situation (for every project after the place this problem was introduced):

  Checked cache for conflicts with source-built nupkgs. Took 00:00:00.0022916
  Finding project directories...
  Finding project.assets.json files...
  Reading usage info...
  Associating usage info with projects...
  Writing data to '/home/dagood/sb/source-build/bin/conflict-report/before-coreclr/usage.json'...
/home/dagood/sb/source-build/repos/dir.targets(202,5): warning : Detected packages in the cache that should be source-built, but aren't. See /home/dagood/sb/source-build/bin/conflict-report/before-coreclr/ for usage details. Newtonsoft.Json 9.0.1 [/home/dagood/sb/source-build/repos/coreclr.proj]

/home/dagood/sb/source-build/bin/conflict-report/before-javascriptservices/usage.json is:

{
  "UnusedPackages": [],
  "Usages": [
    {
      "ProjectDirectory": "/home/dagood/sb/source-build/src/common",
      "AssetsFile": "/home/dagood/sb/source-build/src/common/test/Microsoft.Extensions.CommandLineUtils.Tests/obj/project.assets.json",
      "PackageIdentity": "Newtonsoft.Json/9.0.1",
      "Annotation": null
    }
  ]
}

It does this for every project, since I think it's ok to let the build continue (in this case, at least), and I think it could be interesting to look at each later project's usages (if any).

[dagood]

@dotnet-bot test OSX10.12 Release
Usual ClosedChannelException flakiness.

[dagood]

Interesting, the conflict doesn't happen in the offline tarball build. Target "CheckSourceBuiltNupkgConflictUsages" skipped, due to false condition; ('@(ConflictingPackageInfos)' != '') was evaluated as ('' != ''). Makes sense, since it can't get the online one, and we specifically remove source-built packages from the tarball prebuilt dir.