Skip to content

Re-enable target-based dependency resolution #3121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

Re-enable target-based dependency resolution #3121

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4b6798f
Re-enable target-based dependency resolution
neonichu Dec 16, 2020
0771944
Adjusts tests to latest pubgrub messages
neonichu Dec 16, 2020
2a9acfa
Remove integration test which cannot work with target-based dependenc…
neonichu Dec 18, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 0 additions & 19 deletions IntegrationTests/Tests/IntegrationTests/XCBuildTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,25 +94,6 @@ final class XCBuildTests: XCTestCase {
XCTAssertFileExists(releasePath.appending(component: "bar"))
XCTAssertNoSuchPath(releasePath.appending(component: "cbar"))
}

fixture(name: "XCBuild/ExecutableProducts") { path in
let fooPath = path.appending(component: "Foo")
let binaryPath = fooPath.appending(components: ".build", "apple", "Products")

try sh(swiftBuild, "--package-path", fooPath, "--build-system", "xcode", "--target", "cbar")
let debugPath = binaryPath.appending(component: "Debug")
XCTAssertNoSuchPath(debugPath.appending(component: "foo"))
XCTAssertNoSuchPath(debugPath.appending(component: "cfoo"))
XCTAssertNoSuchPath(debugPath.appending(component: "bar"))
XCTAssertFileExists(debugPath.appending(component: "cbar"))

try sh(swiftBuild, "--package-path", fooPath, "--build-system", "xcode", "--target", "cbar", "-c", "release")
let releasePath = binaryPath.appending(component: "Release")
XCTAssertNoSuchPath(releasePath.appending(component: "foo"))
XCTAssertNoSuchPath(releasePath.appending(component: "cfoo"))
XCTAssertNoSuchPath(releasePath.appending(component: "bar"))
XCTAssertFileExists(releasePath.appending(component: "cbar"))
}
}

func testTestProducts() throws {
Expand Down
4 changes: 0 additions & 4 deletions Sources/PackageGraph/DependencyResolutionNode.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,6 @@ public enum DependencyResolutionNode {

/// Assembles the product filter to use on the manifest for this node to determine its dependencies.
public var productFilter: ProductFilter {
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
switch self {
case .empty:
return .specific([])
Expand All @@ -87,9 +86,6 @@ public enum DependencyResolutionNode {
case .root:
return .everything
}
#else
return .everything
#endif
}

/// Returns the dependency that a product has on its own package, if relevant.
Expand Down
1 change: 1 addition & 0 deletions Sources/PackageGraph/RepositoryPackageContainer.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,7 @@ public class RepositoryPackageContainer: PackageContainer, CustomStringConvertib
productFilter: ProductFilter
) throws -> (Manifest, [Constraint]) {
let manifest = try self.loadManifest(at: revision, version: version)
let productFilter = manifest.toolsVersion < ToolsVersion.v5_2 ? ProductFilter.everything : productFilter
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit of a blunt instrument, but it should yield correct results.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This single line fixes the top secret errors that were reported?

I’m curious if the errors occur generally during fresh resolution, or if they only occur in the presence of a pins file?

I ask because the way I implemented the filtering, manifests < 5.2 asked for every dependency product from every package. Essentially that meant any package that might contain the intended dependency would be resolved. The intent was that if a 5.1 manifest depends on a package that later gets a minor version bump with a 5.2 manifest, that transitive 5.2 manifest knows which of its products were actually requested and can therefore meaningfully filter its own dependencies. What occurs to me know is that had a side effect of culling some dependencies in a way that is logically valid, but not stable compared to previous treatment of 5.2 manifests.

For example, consider the case of a 5.2 manifest which declares a slew of package dependencies, but its actual targets declare no dependencies whatsoever. Under a 5.1 toolchain, all those dependencies were resolved. Under the initial filtering algorithm, all those dependencies were ignored. Ignoring them is valid as far as the build process is concerned. However, because it produces different pins, it introduces an instability where old and new toolchains would each be insisting that the other’s pins are invalid.

Can you look into it to see if this seems to be the real problem, or at least if it is the deciding factor that channels execution into the problematic code branch. If so, then the correct solution is to change the handling of pre‐5.2 manifests to request absolutely everything absolutely all the time.

Such a fix would be done most cleanly here, right where it originates, by injecting something like this:

if toolsVersion < ToolsVersion.v5_2 {
  return dependencies.map { $0.filtered(by: .everything) }
}

From that simple change, everything else should cascade back into place such that manifests from 5.1 and earlier are always and everywhere treated exactly as they were by the 5.1 toolchain. It would mean some graph pruning opportunities would missed, but it would ensure pin stability. Those missed optimizations would only effect old manifests, which presumably will become less and less common over time anyway. (5.2 manifests and above would still be properly skipping everything that isn’t used.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue I have been seeing is that we can miss dependencies of 5.1 and earlier packages that are actually required, the scenario is the same package being required by multiple different packages transitively. I need to dig in a bit more, I put this up more as a straw man so that we have some solution to reactivate target-based dependency resolution that we can land in 5.4.

Where I am at right now, I'm seeing that we don't download a few dependencies as part of resolution which leads us to not being able to include them later on here. This piece of code in Workspace is kind of subtle, because unless we have previously fetched a given package, we'll just fail and ignore it. I'll report back once I have more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to dig in a bit more, I put this up more as a straw man so that we have some solution to reactivate target-based dependency resolution that we can land in 5.4.

Yes, that makes sense.

I still think moving the workaround to the Manifest type like I suggested would be more consistent. I don’t think RepositoryPackageContainer should alter the resolution strategy, as it won’t apply to other conformers of the PackageContainer protocol.

the scenario is the same package being required by multiple different packages transitively

That sounds like there is another place in need of a similar fix to this part of #3006. If a topologicalSort or similar graph traversal uses a node definition of just the package—instead of the package–filter pair—then it would think it had already visited that node and neglect to look at its dependencies even if the filter is different the second time the node shows up. The result would be exactly the symptoms you describe.

return (manifest, manifest.dependencyConstraints(productFilter: productFilter, mirrors: mirrors))
}

Expand Down
8 changes: 0 additions & 8 deletions Sources/PackageModel/Manifest.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,6 @@ public final class Manifest: ObjectIdentifierProtocol {

/// Returns the targets required for a particular product filter.
public func targetsRequired(for productFilter: ProductFilter) -> [TargetDescription] {
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
// If we have already calcualted it, returned the cached value.
if let targets = _requiredTargets[productFilter] {
return targets
Expand All @@ -147,9 +146,6 @@ public final class Manifest: ObjectIdentifierProtocol {
_requiredTargets[productFilter] = targets
return targets
}
#else
return self.targets
#endif
}

/// Returns the package dependencies required for a particular products filter.
Expand Down Expand Up @@ -215,7 +211,6 @@ public final class Manifest: ObjectIdentifierProtocol {
}

return dependencies.compactMap { dependency in
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
if let filter = associations[dependency.name] {
return dependency.filtered(by: filter)
} else if keepUnused {
Expand All @@ -225,9 +220,6 @@ public final class Manifest: ObjectIdentifierProtocol {
// Dependencies known to not have any relevant products are discarded.
return nil
}
#else
return dependency.filtered(by: .everything)
#endif
}
}

Expand Down
12 changes: 0 additions & 12 deletions Tests/BuildTests/BuildPlanTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1314,25 +1314,18 @@ final class BuildPlanTests: XCTestCase {
]
)

#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
XCTAssertEqual(diagnostics.diagnostics.count, 1)
let firstDiagnostic = diagnostics.diagnostics.first.map({ $0.message.text })
XCTAssert(
firstDiagnostic == "dependency 'C' is not used by any target",
"Unexpected diagnostic: " + (firstDiagnostic ?? "[none]")
)
#endif

let graphResult = PackageGraphResult(graph)
graphResult.check(reachableProducts: "aexec", "BLibrary")
graphResult.check(reachableTargets: "ATarget", "BTarget1")
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
graphResult.check(products: "aexec", "BLibrary")
graphResult.check(targets: "ATarget", "BTarget1")
#else
graphResult.check(products: "BLibrary", "bexec", "aexec", "cexec")
graphResult.check(targets: "ATarget", "BTarget1", "BTarget2", "CTarget")
#endif

let planResult = BuildPlanResult(plan: try BuildPlan(
buildParameters: mockBuildParameters(),
Expand All @@ -1341,13 +1334,8 @@ final class BuildPlanTests: XCTestCase {
fileSystem: fileSystem
))

#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
planResult.checkProductsCount(2)
planResult.checkTargetsCount(2)
#else
planResult.checkProductsCount(4)
planResult.checkTargetsCount(4)
#endif
}

func testReachableBuildProductsAndTargets() throws {
Expand Down
7 changes: 0 additions & 7 deletions Tests/PackageGraphTests/PackageGraphTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -739,9 +739,7 @@ class PackageGraphTests: XCTestCase {

DiagnosticsEngineTester(diagnostics) { result in
result.check(diagnostic: "dependency 'Baz' is not used by any target", behavior: .warning)
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
result.check(diagnostic: "dependency 'Biz' is not used by any target", behavior: .warning)
#endif
}
}

Expand Down Expand Up @@ -1080,11 +1078,6 @@ class PackageGraphTests: XCTestCase {
}

func testUnreachableProductsSkipped() throws {
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
#else
try XCTSkipIf(true)
#endif

let fs = InMemoryFileSystem(emptyFiles:
"/Root/Sources/Root/Root.swift",
"/Immediate/Sources/ImmediateUsed/ImmediateUsed.swift",
Expand Down
5 changes: 0 additions & 5 deletions Tests/PackageGraphTests/PubgrubTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1062,11 +1062,6 @@ final class PubgrubTests: XCTestCase {
}

func testUnreachableProductsSkipped() throws {
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
#else
try XCTSkipIf(true)
#endif

builder.serve("root", at: .unversioned, with: [
"root": ["immediate": (.versionSet(v1Range), .specific(["ImmediateUsed"]))]
])
Expand Down
11 changes: 2 additions & 9 deletions Tests/PackageGraphTests/RepositoryPackageContainerProviderTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -357,11 +357,6 @@ class RepositoryPackageContainerProviderTests: XCTestCase {
}

func testDependencyConstraints() throws {
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
#else
try XCTSkipIf(true)
#endif

let dependencies = [
PackageDependencyDescription(name: "Bar1", url: "/Bar1", requirement: .upToNextMajor(from: "1.0.0")),
PackageDependencyDescription(name: "Bar2", url: "/Bar2", requirement: .upToNextMajor(from: "1.0.0")),
Expand Down Expand Up @@ -627,10 +622,8 @@ class RepositoryPackageContainerProviderTests: XCTestCase {

let forNothing = try container.getDependencies(at: version, productFilter: .specific([]))
let forProduct = try container.getDependencies(at: version, productFilter: .specific(["Product"]))
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
// If the cache overlaps (incorrectly), these will be the same.
XCTAssertNotEqual(forNothing, forProduct)
#endif
// If the cache overlaps (incorrectly), these will be the same.
XCTAssertNotEqual(forNothing, forProduct)
}
}
}
Expand Down
4 changes: 0 additions & 4 deletions Tests/PackageModelTests/ManifestTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,11 @@ class ManifestTests: XCTestCase {
targets: targets
)

#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
XCTAssertEqual(manifest.targetsRequired(for: .specific(["Foo", "Bar"])).map({ $0.name }).sorted(), [
"Bar",
"Baz",
"Foo",
])
#endif
}
}

Expand Down Expand Up @@ -152,13 +150,11 @@ class ManifestTests: XCTestCase {
targets: targets
)

#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
XCTAssertEqual(manifest.dependenciesRequired(for: .specific(["Foo"])).map({ $0.name }).sorted(), [
"Bar1", // Foo → Foo1 → Bar1
"Bar2", // Foo → Foo1 → Foo2 → Bar2
// (Bar3 is unreachable.)
])
#endif
}
}
}
17 changes: 2 additions & 15 deletions Tests/WorkspaceTests/WorkspaceTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1524,11 +1524,7 @@ final class WorkspaceTests: XCTestCase {
// Run update.
workspace.checkUpdateDryRun(roots: ["Root"]) { changes, diagnostics in
XCTAssertNoDiagnostics(diagnostics)
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
let stateChange = Workspace.PackageStateChange.updated(.init(requirement: .version(Version("1.5.0")), products: .specific(["Foo"])))
#else
let stateChange = Workspace.PackageStateChange.updated(.init(requirement: .version(Version("1.5.0")), products: .everything))
#endif

let path = AbsolutePath("/tmp/ws/pkgs/Foo")
let expectedChange = (
Expand Down Expand Up @@ -2089,13 +2085,11 @@ final class WorkspaceTests: XCTestCase {
]
)

#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
workspace.checkPackageGraph(roots: ["Root"]) { _, diagnostics in
DiagnosticsEngineTester(diagnostics) { result in
result.check(diagnostic: .contains("Foo[Foo] 1.0.0..<2.0.0"), behavior: .error)
result.check(diagnostic: .contains("no versions of 'Foo' match the requirement 1.0.1..<2.0.0 and 'Foo' 1.0.0 contains incompatible tools version"), behavior: .error)
}
}
#endif
}

func testToolsVersionRootPackages() throws {
Expand Down Expand Up @@ -2928,11 +2922,9 @@ final class WorkspaceTests: XCTestCase {
result.check(packages: "Foo")
result.check(targets: "Foo")
}
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
DiagnosticsEngineTester(diagnostics) { result in
result.check(diagnostic: .contains("Bar[Bar] {1.0.0..<1.5.0, 1.5.1..<2.0.0} is forbidden"), behavior: .error)
result.check(diagnostic: .contains("package 'bar' is required using a stable-version but 'bar' depends on an unstable-version package 'baz'."), behavior: .error)
}
#endif
}
}

Expand Down Expand Up @@ -4065,11 +4057,6 @@ final class WorkspaceTests: XCTestCase {
}

func testTargetBasedDependency() throws {
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
#else
try XCTSkipIf(true)
#endif

let sandbox = AbsolutePath("/tmp/ws/")
let fs = InMemoryFileSystem()

Expand Down
14 changes: 0 additions & 14 deletions Tests/XCBuildSupportTests/PIFBuilderTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,20 +86,11 @@ class PIFBuilderTests: XCTestCase {
let targetAExeDependencies = pif.workspace.projects[0].targets[0].dependencies
XCTAssertEqual(targetAExeDependencies.map{ $0.targetGUID }, ["PACKAGE-PRODUCT:blib", "PACKAGE-TARGET:A2", "PACKAGE-TARGET:A3"])
let projectBTargetNames = pif.workspace.projects[1].targets.map({ $0.name })
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
XCTAssertEqual(projectBTargetNames, ["blib", "B2"])
#else
XCTAssertEqual(projectBTargetNames, ["bexe", "blib", "B2"])
#endif
}
}

func testProject() throws {
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
#else
try XCTSkipIf(true)
#endif

let fs = InMemoryFileSystem(emptyFiles:
"/Foo/Sources/foo/main.swift",
"/Foo/Tests/FooTests/tests.swift",
Expand Down Expand Up @@ -685,11 +676,6 @@ class PIFBuilderTests: XCTestCase {
}

func testTestProducts() throws {
#if ENABLE_TARGET_BASED_DEPENDENCY_RESOLUTION
#else
try XCTSkipIf(true)
#endif

let fs = InMemoryFileSystem(emptyFiles:
"/Foo/Sources/FooTests/FooTests.swift",
"/Foo/Sources/CFooTests/CFooTests.m",
Expand Down