Index targets top-down instead of bottom up#2684
Conversation
| var sortedTargets: [BuildTargetIdentifier] = | ||
| await orLog("Sorting targets") { try await buildServerManager.topologicalSort(of: Array(filesByTarget.keys)) } | ||
| await orLog("Sorting targets") { | ||
| try await buildServerManager.targetsSortedForIndexing(Array(filesByTarget.keys)) |
There was a problem hiding this comment.
If batchSize is greater than 1, the targets end up being partitioned in a rather odd way. I don't believe preparationBatchingStrategy is widely used, but this change puts it in a somewhat awkward position.
There was a problem hiding this comment.
Why do you think it sorts them in an odd way? It shouldn’t be any more odd than it is today, right?
|
@swift-ci Please test |
| // Now walk through the list of targets in the work list. For each target from the work list, index all of its | ||
| // transitive dependencies next. We do this because preparing a top-level target likely also prepared all of its |
There was a problem hiding this comment.
Hmm this does feel like it's undermining the previous sorting a bit, since now this means we'll potentially index files from dependency packages before files in the root package, right? Should we be maintaining the root/dependency package split?
There was a problem hiding this comment.
Yes, it does. But in practice I have found that the bottleneck to indexing is exclusively target preparation. So, if we chose to not index these files, a processor core would likely just be sitting idle. Furthermore, since all the preparation tasks are scheduled before the file index tasks, we should always prefer a target preparation over file indexing.
| await orLog("Preparing") { try await prepare(singleTarget: target) } | ||
| await orLog("Preparing") { | ||
| try await prepare(singleTarget: target) | ||
| implicitlyPreparedTargets += transitiveClosure(of: [target], successors: { targetDependencies[$0] ?? [] }) |
There was a problem hiding this comment.
Do we have to worry about BSP servers that may choose a different platform for the target vs one of its dependencies? e.g
A (iOS) -> B (iOS + macOS)
If the current run destination is macOS and we try to prepare A, a BSP might reasonably decide to fallback and prepare it using iOS instead, since that's the only thing it supports. B then also gets prepared for iOS. But if you ask for preparation of B with a macOS run destination, naturally you'd expect macOS.
cc @AnthonyLatsis who might also have thoughts here
There was a problem hiding this comment.
Targets in BSP are always configured targets for one particular platform. So we can’t have a target that is for multiple platforms here.
We originally started preparing and indexing the low-level targets in the hope that we can unblock the files of those targets as early as possible for indexing and thus get best indexing performance. Revisiting this design decision, we can actually get a ~2x speed up (SourceKit-LSP gets indexed in 2:45 instead of 5:49 minutes on my machine) in initial indexing performance by preparing indexing top-level targets first. This speed-up is for two reasons: - When preparing a top-level target, all of its dependencies are implicitly also prepared. That means that we don’t need to send any prepare requests to the build server for those targets, which is very advantages for SwiftPM projects, in which the constant cost in the order of 500ms (based on my memory) even for a null build. - The underlying build server is able to prepare dependencies of the top-level target in parallel instead of implicitly serializing them by preparing all targets in the package in dependency order.
ahoppen
force-pushed
the
top-down-indexing
branch
from
cd5f548 to
af0452e
Compare
3 participants
We originally started preparing and indexing the low-level targets in the hope that we can unblock the files of those targets as early as possible for indexing and thus get best indexing performance. Revisiting this design decision, we can actually get a ~2x speed up (SourceKit-LSP gets indexed in 2:45 instead of 5:49 minutes on my machine) in initial indexing performance by preparing indexing top-level targets first. This speed-up is for two reasons: