generator: The target toolchain is still needed to build a package-based SDK

PR swiftlang#177 was intended to skip unnecessary toolchain downloads, making it
possible to build a container-based SDK offline if all its requirements
had already been dowloaded.  The change was too broad and also broke
building package-based SDKs.

PR swiftlang#177 skips calling generator.downloadArtifacts() when building
an SDK without an embedded host toolchain (the default).   In
addition to downloading the host toolchain (and LLVM, if needed),
generator.downloadArtifacts() is also responsible for downloading
the target toolchain.  This is not needed when building a container-based
SDK but is required when building a package-based SDK, which combines
an SDK from swift.org with supporting libraries extracted from
Debian packages.

In fact, generator.downloadArtifacts() already avoids downloading
toolchains when building a container-based SDK without an embedded
toolchain.   The only network call which caused offline builds to
fail was an unconditional check for a suitable LLVM binary from
GitHub.

This PR restores the call to generator.downloadArtifacts() and only
makes the LLVM check if LLVM is in the list of required downloads.
This allows the EndToEnd tests to pass again (with PR swiftlang#170 temporarily
reverted because of issue swiftlang#181).

Author: euanh

Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator+Download.swift

@@ -39,15 +39,18 @@ extension SwiftSDKGenerator {
     logger.info("Downloading required toolchain packages...")
     let hostLLVMURL = downloadableArtifacts.hostLLVM.remoteURL
     // Workaround an issue with github.com returning 400 instead of 404 status to HEAD requests from AHC.
-    let isLLVMBinaryArtifactAvailable = try await type(of: client).with(http1Only: true) {
-      try await $0.head(
-        url: hostLLVMURL.absoluteString,
-        headers: ["Accept": "*/*", "User-Agent": "Swift SDK Generator"]
-      )
-    }
 
-    if !isLLVMBinaryArtifactAvailable {
-      downloadableArtifacts.useLLVMSources()
+    if itemsToDownload(downloadableArtifacts).contains(where: { $0.remoteURL == downloadableArtifacts.hostLLVM.remoteURL } ) {
+      let isLLVMBinaryArtifactAvailable = try await type(of: client).with(http1Only: true) {
+        try await $0.head(
+          url: hostLLVMURL.absoluteString,
+          headers: ["Accept": "*/*", "User-Agent": "Swift SDK Generator"]
+        )
+      }
+
+      if !isLLVMBinaryArtifactAvailable {
+        downloadableArtifacts.useLLVMSources()
+      }
     }
 
     let results = try await withThrowingTaskGroup(of: FileCacheRecord.self) { group in

Sources/SwiftSDKGenerator/SwiftSDKRecipes/LinuxRecipe.swift

@@ -236,14 +236,12 @@ public struct LinuxRecipe: SwiftSDKRecipe {
       generator.pathsConfiguration
     )
 
-    if hostSwiftSource != .preinstalled {
-        try await generator.downloadArtifacts(
-          client,
-          engine,
-          downloadableArtifacts: &downloadableArtifacts,
-          itemsToDownload: { artifacts in itemsToDownload(from: artifacts) }
-        )
-    }
+    try await generator.downloadArtifacts(
+      client,
+      engine,
+      downloadableArtifacts: &downloadableArtifacts,
+      itemsToDownload: { artifacts in itemsToDownload(from: artifacts) }
+    )
 
     if !self.shouldUseDocker {
       guard case let .ubuntu(version) = linuxDistribution else {