[Autolink] Autolinking on COFF for Cygwin/MinGW

[tinysun212]

What's in this pull request?

Resolved bug number: (SR-1128)

---

Before merging this pull request to apple/swift repository:

• Test pull request on Swift continuous integration.

Triggering Swift CI

The swift-ci is triggered by writing a comment on this PR addressed to the GitHub user @swift-ci. Different tests will run depending on the specific comment that you use. The currently available comments are:

Smoke Testing

Platform	Comment
All supported platforms	@swift-ci Please smoke test
All supported platforms	@swift-ci Please smoke test and merge
OS X platform	@swift-ci Please smoke test OS X platform
Linux platform	@swift-ci Please smoke test Linux platform

A smoke test on macOS does the following:

1. Builds the compiler incrementally.
2. Builds the standard library only for macOS. Simulator standard libraries and
   device standard libraries are not built.
3. lldb is not built.
4. The test and validation-test targets are run only for macOS. The optimized
   version of these tests are not run.

A smoke test on Linux does the following:

1. Builds the compiler incrementally.
2. Builds the standard library incrementally.
3. lldb is built incrementally.
4. The swift test and validation-test targets are run. The optimized version of these
   tests are not run.
5. lldb is tested.

Validation Testing

Platform	Comment
All supported platforms	@swift-ci Please test
All supported platforms	@swift-ci Please test and merge
OS X platform	@swift-ci Please test OS X platform
OS X platform	@swift-ci Please benchmark
Linux platform	@swift-ci Please test Linux platform

Lint Testing

Language	Comment
Python	@swift-ci Please Python lint

Note: Only members of the Apple organization can trigger swift-ci.

Cygwin and MinGW should use the autolink feature in the sameway of Linux
due to the linker's limit. Now swift-autolink-extract recognizes the
COFF format file for Cygwin/MinGW.

[gribozavr]

Please add the autolink-extract feature to your targets in test/lit.cfg, and update the tests in test/AutolinkExtract for the COFF format.

[compnerd]

I still think that we should figure out why you can't use the existing linker support on these targets. LLVM can generate objects which conform to binutils' limitations. If this is a workaround, we should at least have a clear understanding of what it is working around and test for the case that shows what breaks down.

[tinysun212]

(For other readers, some of this discussion is overlapped with #3841, because the compiler part of autolinking is there)

The root cause is to ld not LLVM. The ld does not support autolinking although it can read .drectve section.

We can confirm all the possible .drectve features of ld from its source. (line 875883, 934950)
They are -heap, -stack, -attr, -export, -aligncomm. So -export is possible, but -lFoo is not possible.

[compnerd]

Ah, right, that was a local extension that I had toyed with. Id say that we want to base the decision on the linker -use-ld rather than on the environment. If the environment is using lld, then it should be able to support the linking as well. We definitely should have a test that checks the negative condition.

[tinysun212]

lld has not ported to Cygwin. Currently, it has two problems to replace the ld of Cygwin/MinGW.

1. COFF support ( lld -flavor link ) acts exactly as if it is MS LINK, so it does not support ld style options.
   GNU support ( lld -flavor gnu ) can handle ld style options, but it can process only ELF format.
2. Some features, that ld of Cygwin/MinGW has, are not yet implemented.
   ('using auto-export functionality', 'automatic data imports', 'direct linking to a dll'; Reference: ld and WIN32)

[compnerd]

Right, but the MinGW folks have been playing with lld for Windows, and are expecting to use it because I didn't want to upstream my patches for binutils to support Windows ARM :-). So, I imagine that they will at some point try to converge. Thats much less likely for cygwin, but in both scenarios, the check over the value of -use-ld ensures that the correct behavior is preserved.

[tinysun212]

Is 'MinGW with lld for Windows' possible for MinGW applications which is liking with glibc?
As I know, the COFF support of lld for Windows can't be used for liking glibc.

I'm waiting someone implement it. (llvm-dev: lld for MinGW)

[compnerd]

For the most part, yes, it is possible. The arguments just need to be mapped into the appropriate spelling. There are only a couple of misfeatures which aren't necessary for swift which I can see as being contentious.

[tinysun212]

I now understand your opinion. I agree it is possible to use lld for Windows as a linker by mapping arguments in MinGW. It is possible because any MinGW application is inherently also a Windows application.

But we can not use the lld for autolinking.

In MinGW, gcc/clang and ld uses the '.drective' convention which is listing the linker options starting with '-' and limited to five options. To use 'lld for Windows', we should put the options starting with '/DEFAULTLIB:' to the '.drective' section. This violates the convention for MinGW object file and lead to generate warnings whenever native linker ld links the object with other normal object file.

[compnerd]

/defaultlib: has nothing to do with the object format. It is purely a linker directive, and does not violate any requirements for MinGW. We could actually add support for the -l<library> option pretty easily to lld though, it seems better to just use -defaultlib: as the spelling. Supporting -defaultlib in BFD is pretty trivial in fact.

[tinysun212]

Does swift project have a plan for covering its own linker?

If we now put -l<library> to .drective section, it generates warning messages by the standard linker for MinGW. Isn't it violate?

If you patched the ld and lld for -l<library> feature, then we can add the feature to Swift. But if it is not widely deployed, we should include the our-defaultlib-support-linker into Swift package for MinGW. It is desired that we use only the current widely deployed tools rather than including customizing tool.

The enhance-lld-or-ld-work will takes some time which is not dependent to the patch size, but rather dependent to llvm-lld development community and ld development community.

If we success these (patch ld/lld and widely deployed), we can apply same thing to ELF format with proper section name, and the swift-autolink-extract can be obsoleted.

[tinysun212]

@gribozavr, I updated test related files.

[gribozavr]

Would you mind using a more compact form?

elif run_os in ['linux-gnu', 'linux-gnueabihf', 'freebsd', 'windows-cygnus', 'windows-gnu']:

[gribozavr]

@tinysun212 Thank you, this looks great! Only a few minor comments, and we should be good to go!

Let's get a CI run anyway:

@swift-ci Please test

[tinysun212]

@gribozavr, I applied your comments.

[gribozavr]

@swift-ci Please test

[compnerd]

This isn't valid for COFF. COFF has a 8-character limit on section names.

[tinysun212]

I quoted the description of the name field in 3. Section Table (Section Headers) from MS PECOFF.

... For longer names, this field contains a slash (/) that is followed by an ASCII 
representation of a decimal number that is an offset into the string table.
Executable images do not use a string table and do not support section names longer
than 8 characters. Long names in object files are truncated if they are emitted to 
an executable file.

Because the section '.swift1_autolink_entries' will not be emitted to an executable file, we can use the long name for autolink feature. But we can not use the long names '.swift2_protocol_conformances' and '.swift2_type_metadata' instead of '.sw2prtc' or '.sw2tymd', because they are emitted to an executable and loaded into the system memory.

[gribozavr]

@swift-ci Please test Linux platform

[gribozavr]

Testing Linux manually.

[gribozavr]

Tests on Linux passed. @tinysun212 Would you mind submitting the simplification around llvm::dyn_cast as a separate PR? I'd like to merge this one.