Fix direct clang cc1 emit-pcm commands with vfs overlay on Windows #85325

hjyamauchi · 2025-11-05T01:35:18Z

Explicit module builds currently fail on Windows because direct-clang-cc1-module-build emit-pcm commands take overlaid system module map files as inputs but miss the clang VFS overlay. This change adds the overlay and fixes explicit module builds on Windows.

hjyamauchi · 2025-11-05T01:35:33Z

@swift-ci please test

owenv · 2025-11-05T01:44:42Z

nice, I expect this might let us remove some of the Windows skips in swiftlang/swift-build#861, I was looking at some failures there due to a missing VFS post-scanning

cachemeifyoucan · 2025-11-06T17:30:22Z

lib/Frontend/Frontend.cpp

  CDP->startDiagnosticCapture();
 }

+void CompilerInstance::setUpVFSForDirectClangCC1EmitPCM() {


Why is this not in setUpVirtualFileSystemOverlays?

The reason of the location of the setUpVFSForDirectClangCC1EmitPCM call was because calling applyClangInvocationMapping would require the ASTContext (after setUpASTContextIfNeeded) but needs to before inputs are read (before setUpInputs()). If ASTContext could be set up before setUpVirtualFileSystemOverlays (which I haven't attempted), setUpVFSForDirectClangCC1EmitPCM could have been in setUpVirtualFileSystemOverlays.

I think applyCalngInvocationMapping can probably get away with Invocation + DiagnosticEngine + an allocator instead of ASTContext.

cachemeifyoucan · 2025-11-06T17:32:04Z

lib/Frontend/Frontend.cpp

+        SourceMgr.getFileSystem();
+    ClangInvocationFileMapping FileMapping = applyClangInvocationMapping(
+        *Context, nullptr, VFS, /*suppressDiagnostic=*/false);
+    if (!FileMapping.redirectedFiles.empty()) {


I don't think this is correct for caching. Caching build always uses direct clang cc1 mode but the input is include-tree, which doesn't have any notion of VFS anymore.

I understand generally that in the caching build, inputs are all loaded into CAS (include-trees) by the depscan command and then read from include-trees only by the subsequent build commands. But I am not sure how that maps into the code here or I'm under-informed about the CAS case as I have started looking into (swift) CAS builds yet.

For a caching build, I imagine, direct clang cc1 mode commands receive an an include tree via flags and setUpVirtualFileSystemOverlays will set that up so input module map file (such as vcruntime.modulemap) will be read from the include tree?

For non-caching builds, we'd need something that enables VFS. so perhaps we need to move setUpVFSForDirectClangCC1EmitPCM to setUpVirtualFileSystemOverlays and only conditionally run for non caching builds? But that doesn't match with the following comment "swift-frontend should not touch any inputs and apply any clang argument transformation when compiling".

Sounds correct.

IncludeTree already contains everything needed inside the CAS, including files, module maps, and all the VFS overlay has been flattened into lookup content so never pass a VFS to include tree build.

cachemeifyoucan · 2025-11-06T17:38:08Z

test/ScanDependencies/direct-clang-cc1-emit-pcm-vfs-overlay.swift

+// RUN: %empty-directory(%t)
+
+// Test that the -direct-clang-cc1-module-build is able to create a module from the VC runtime with an overlaid modulemap file
+// RUN: %swift_frontend_plain -frontend -emit-pcm -module-name SAL -o %t/SAL.pcm -direct-clang-cc1-module-build "%vctoolsinstalldir\include\module.modulemap" -sdk %S/Inputs/WinSDK -Xcc -fsystem-module -Xcc -emit-module -Xcc -isystem -Xcc "%vctoolsinstalldir\include" -Xcc -cc1 -Xcc -fmodules -Xcc -fmodules-cache-path=%t/module-cache


This test is very insufficient.

The explicit module build is always a combination of scanner and swift-frontend. There are few different ways to tackle this problem, and I don't think this is the correct way to fix.

The concept of direct cc1 mode is that swift-frontend should not touch any inputs and apply any clang argument transformation when compiling. Everything should be done by dependency scanner. It is dependency scanner's job to construct tasks that contains the correct VFS for swift-frontend to use. I would prefer the fix to be done that way.

For test case, we have to test the combination of scanner and swift-frontend, otherwise putting a bunch of clang cc1 arguments in the tests doesn't really test what we want.

I agree about the test and the overall point; but, one thing that makes this trickier is that these particular VFS overlays and the files they point to (.modulemaps) are conjured up completely in-memory by the compiler.

For compilation, this is done in ClangImporter::create by calling applyClangInvocationMapping, and the scanner does something similar and calls into applyClangInvocationMapping when setting up the Clang scanner filesystem.

So now the scanner constructs invocations that assume that this virtual filesystem is present, but the CC1 task doesn't assemble the same in-memory filesystem during execution, if I understand this correctly.

Caching seems to make this particularly tricky - I don't think I have any ideas on how to fit this into include trees.

Include tree abstract that away already. It just needs to be seen by scanner and you don't need to pass it or add that during compilation.

If the in memory overlay is needed for non-cache build, it is fine. The overlay need not be there for caching build and please add a test case for caching (if that works, if not, we can hold that off for now).

The concept of direct cc1 mode is that swift-frontend should not touch any inputs and apply any clang argument transformation when compiling. Everything should be done by dependency scanner. It is dependency scanner's job to construct tasks that contains the correct VFS for swift-frontend to use. I would prefer the fix to be done that way.

So it sounds like instead of setUpVFSForDirectClangCC1EmitPCM, the vfs overlay somehow needs to be passed to the build (emit-pcm) command probably via flags. I will think about this and come back.

I see that we need the depscan and the build commands in the test.

Based on the above discussion (which I missed before posting my comment), I'll try going down the approach of calling applyClangInvocationMapping in the cc1 (emit-pcm) command for non-caching builds, as it doesn't seem easy to set up applyClangInvocationMapping sets up with a flag from the depscan side. Please let me know if I didn't interpret it right.

hjyamauchi · 2025-11-10T21:34:56Z

@swift-ci please test

artemcm · 2025-11-10T22:20:02Z

lib/Frontend/Frontend.cpp

+      Invocation.getClangImporterOptions().DirectClangCC1ModuleBuild &&
+      Invocation.getFrontendOptions().RequestedAction ==
+          FrontendOptions::ActionType::EmitPCM) {
+    // For non-caching builds, transfer the ownership to the ASTContext.


Suggested change

// For non-caching builds, transfer the ownership to the ASTContext.

// For non-caching direct-cc1 PCM builds, transfer the ownership to the ASTContext.

artemcm · 2025-11-10T22:25:34Z

include/swift/ClangImporter/ClangImporter.h

    llvm::IntrusiveRefCntPtr<llvm::vfs::FileSystem> vfs = nullptr,
    bool suppressDiagnostic = false);

+ClangInvocationFileMapping getClangInvocationFileMapping(


I'd prefer it if we unified the API here so that the client doesn't need to wonder at a glance which one they need to call. If we're committing to being able to do this without an ASTContext, then we should stick to only having the general version.

It feels to me like the one without an ASTContext is more of a special version because it is limited to this particular case before ASTContext is created and is more complicated to have an ownership transfer.

Done. There is only one version now

artemcm · 2025-11-10T23:50:45Z

include/swift/AST/ASTContext.h

  /// The block list where we can find special actions based on module name;
  BlockListStore blockListConfig;
+
+  llvm::BumpPtrAllocator ClangInvocationFileMappingAllocator;


It does not appear that this gets used? In which case the logic of transferring the corresponding allocator from the CompilerInstance to here is dubious.

This is used in Frontend.cpp:373 as the destination of the transfer/mov near the above comment suggestion

Is the ASTContext's assigned ClangInvocationFileMappingAllocator ever used other than as a destination of the transfer in Frontend.cpp?

I mean to put it there so that the allocator has the same lifetime as the ASTContext and is to be destructed when the ASTContext is destructed (the ASTContext's destructor uses it )

cachemeifyoucan · 2025-11-11T00:46:40Z

lib/ClangImporter/ClangIncludePaths.cpp

                     << "' with the following contents:\n";
        llvm::errs() << file.second << "\n";
      }
      auto contents = ctx.Allocate<char>(file.second.size() + 1);


I don't know why does this need to be allocated in ASTContext? OverlayFileSystem is taking the ownership of the buffer so why do we need a copy in ASTContext?

My reading is that OverlayFileSystem doesn't currently take the owership of the buffer as the getMemBuffer takes a StringRef, suggesting a non-owing buffer, but if we use getMemBufferCopy instead, I think it will and with it we won't need to worry about an allocator. I'll test this.

This worked.

cachemeifyoucan

I am not quite sure about the allocator. Other than that, it is fine conceptually.

cachemeifyoucan · 2025-11-11T00:49:38Z

lib/Frontend/Frontend.cpp

      Invocation.getLangOptions().DebuggerSupport)
    Invocation.getLangOptions().EnableDeserializationSafety = false;

+  if (!Invocation.getCASOptions().requireCASFS() &&


I will just check EnableCaching rather than requireCASFS().

Explicit module builds currently fail on Windows because direct-clang-cc1-module-build emit-pcm commands take overlaid system module map files as inputs but miss the clang VFS overlay. This change adds the overlay and fixes explicit module builds on Windows.

hjyamauchi · 2025-11-12T19:02:14Z

@swift-ci please test

cachemeifyoucan

Much cleaner. LGTM.

artemcm

Thank you, @hjyamauchi !

…wiftlang#85325) Explicit module builds currently fail on Windows because direct-clang-cc1-module-build emit-pcm commands take overlaid system module map files as inputs but miss the clang VFS overlay. This change adds the overlay and fixes explicit module builds on Windows.

hjyamauchi marked this pull request as ready for review November 6, 2025 17:26

hjyamauchi requested review from artemcm, cachemeifyoucan and tshortli as code owners November 6, 2025 17:26

cachemeifyoucan requested changes Nov 6, 2025

View reviewed changes

hjyamauchi force-pushed the emb-1 branch from 38a998a to 7b4835f Compare November 10, 2025 21:32

hjyamauchi requested review from Xazax-hun, beccadax, egorzhdan, hborla, ian-twilightcoder, j-hui, slavapestov, susmonteiro and xedin as code owners November 10, 2025 21:32

artemcm reviewed Nov 10, 2025

View reviewed changes

cachemeifyoucan reviewed Nov 11, 2025

View reviewed changes

hjyamauchi force-pushed the emb-1 branch from 7b4835f to dca3538 Compare November 12, 2025 19:00

cachemeifyoucan approved these changes Nov 12, 2025

View reviewed changes

artemcm approved these changes Nov 12, 2025

View reviewed changes

hjyamauchi merged commit a96b57d into swiftlang:main Nov 13, 2025
5 checks passed

hjyamauchi mentioned this pull request Nov 13, 2025

error: no type named '_GUID' in module '_GUIDDef in WinSDK with C++ interop and explicit-module-build #85495

Open

	// For non-caching builds, transfer the ownership to the ASTContext.
	// For non-caching direct-cc1 PCM builds, transfer the ownership to the ASTContext.

Fix direct clang cc1 emit-pcm commands with vfs overlay on Windows #85325

Fix direct clang cc1 emit-pcm commands with vfs overlay on Windows #85325

Conversation

hjyamauchi commented Nov 5, 2025

Uh oh!

hjyamauchi commented Nov 5, 2025

Uh oh!

owenv commented Nov 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hjyamauchi Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hjyamauchi commented Nov 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hjyamauchi Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hjyamauchi Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cachemeifyoucan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hjyamauchi commented Nov 12, 2025

Uh oh!

cachemeifyoucan left a comment

Choose a reason for hiding this comment

Uh oh!

artemcm left a comment

Choose a reason for hiding this comment

Uh oh!

hjyamauchi Nov 6, 2025 •

edited

Loading

hjyamauchi Nov 11, 2025 •

edited

Loading

hjyamauchi Nov 11, 2025 •

edited

Loading