3C documentation update pass 2021-02 (#438)

Author: mattmccutchen-cci

clang/docs/checkedc/3C/CONTRIBUTING.md

@@ -1,9 +1,8 @@
 # Contributing to 3C
 
 Issues and pull requests related to 3C should be submitted to [CCI's
-repository](https://github.com/correctcomputation/checkedc-clang), not
-[Microsoft's Checked C
-repository](https://github.com/microsoft/checkedc-clang), except as
+`checkedc-clang` repository](https://github.com/correctcomputation/checkedc-clang), not
+[Microsoft's](https://github.com/microsoft/checkedc-clang), except as
 stated below.
 
 ## Issues
@@ -30,19 +29,20 @@ your version.
 
 If your contribution does not touch any 3C-specific code (or is a
 codebase-wide cleanup of low risk to 3C) and you can reasonably submit
-it to [Microsoft's Checked C
+it to [Microsoft's
 repository](https://github.com/microsoft/checkedc-clang) instead, we
 generally prefer that you do so. If such a contribution has particular
 benefit to 3C, feel free to let us know, and we may assist you in
 getting your contribution accepted upstream and/or ensure it is merged
 quickly to CCI's repository.
 
 If the previous paragraph does not apply, just submit a pull request
-here. You must grant the same license on your contribution as the
-existing codebase. We do not have a formal contributor license
-agreement (CLA) process at this time, but we may set one up and
-require you to complete it before we accept your contribution. Also be
-aware that we need to keep 5C ([our proprietary extension of
+to CCI's repository. You must grant the same license on your
+contribution as the existing codebase. We do not have a formal
+contributor license agreement (CLA) process at this time, but we may
+set one up and require you to complete it before we accept your
+contribution. Also be aware that we need to keep 5C ([our proprietary
+extension of
 3C](README.md#what-3c-users-should-know-about-the-development-process))
 working, so you may have to wait for us to address 5C-specific
 problems arising from your 3C pull request and/or we may ask you to
@@ -58,7 +58,7 @@ an idea.) The easiest way to run it is to run the following in your
 build directory:
 
 ```
-ninja check-clang-3c
+ninja check-3c
 ```
 
 This command will build everything needed that hasn't already been
@@ -67,6 +67,32 @@ you can use it in scripts), and display some information about any
 failures, which may or may not be enough for you to understand what
 went wrong.
 
+For deeper troubleshooting, run the following in your build directory
+to build all dependencies of the test suite:
+
+```
+ninja check-3c-deps
+```
+
+Then run the following in the `clang/test/3C` directory:
+
+```
+llvm-lit -vv TEST.c
+```
+
+where `TEST.c` is the path of the test you want to run (you can also
+specify more than one test). This assumes you've put the `bin`
+subdirectory of your build directory on your `$PATH` or arranged some
+other means of running `llvm-lit` from there. The first `-v` makes
+`llvm-lit` display the stdout and stderr of failed tests; the second
+makes it display the `RUN` commands as they execute so you can tell
+which one failed.
+
+Every `.c` file under `clang/test/3C` is a test file. There are a few
+in subdirectories, so `*.c` will not pick up all of them; instead you
+can use `llvm-lit -vv .` to specify all test files under the current
+directory.
+
 ## Coding guidelines
 
 Please follow [LLVM coding
@@ -88,6 +114,10 @@ in your code. Specifically:
 
 * Space before and after `:` in iterators, i.e., `for (auto &k : List)`
 
-All files should be formatted with `clang-format` and pass `clang-tidy` ([more
-information](clang-tidy.md)), and nonempty files should have a final newline
-(surprisingly, `clang-format` cannot enforce this).
+Our goal is that all files should be formatted with `clang-format` and
+pass `clang-tidy` ([more information](clang-tidy.md)), and nonempty
+files should have a final newline (surprisingly, `clang-format` cannot
+enforce this). However, until we have better automation, we decided it
+isn't reasonable to require contributors to manually run these tools
+and fix style nits in each change; instead, we periodically run the
+tools on the entire 3C codebase.

clang/docs/checkedc/3C/INSTALL.md

@@ -9,32 +9,45 @@ documentation as appropriate.
 
 ## Supported environments
 
-We work primarily on Linux. Checked C supports Windows, but we do not
-test 3C regularly on Windows. Some of us use Mac OS X, which Checked C
-does not officially support; we have gotten 3C to mostly work with
-some additional steps (see below), but there are some known problems
-and [Microsoft is still considering whether to work with us to address
-them](https://github.com/microsoft/checkedc/issues/424#issuecomment-720648442).
+We aim to support Linux, Windows, and Mac OS X. We do most of our work
+on Linux and test the other OSes less frequently ([additional
+information](README.md#which-checkedc-clang-repository-to-use)), so
+you are more likely to run into problems on the other OSes; if you do,
+please [file an issue](CONTRIBUTING.md#issues) and we will try to fix
+it if it is severe enough. Getting 3C to work on Mac OS X [can be
+fiddly](#mac-os-x), but we have managed it so far.
 
 ## Basics
 
-Assuming you have already cloned the `checkedc-clang` repository, run
-the following (from the root of the working tree) for a basic build:
+As described [here](README.md#which-checkedc-clang-repository-to-use),
+you can use either
+https://github.com/correctcomputation/checkedc-clang or
+https://github.com/microsoft/checkedc-clang (or, of course, a
+third-party fork, though we can't be responsible for that). Assuming
+you have already cloned one of these repositories, run the following
+(from the root of the working tree) for a basic build:
 
 ```
-(cd llvm/projects/checkedc-wrapper && git clone https://github.com/microsoft/checkedc)
+# Get a copy of the Checked C system headers. Use Microsoft's
+# "checkedc" repository regardless of which "checkedc-clang"
+# repository you use.
+git clone https://github.com/microsoft/checkedc llvm/projects/checkedc-wrapper/checkedc
 
 mkdir build && cd build
 cmake ../llvm -G Ninja -DLLVM_ENABLE_PROJECTS=clang
 ninja TARGET
 ```
 
-where TARGET stands for the target(s) you want to build. For the `3c`
-command-line tool, the target is `3c`, but assuming you want to
+where `TARGET` stands for the target(s) you want to build. For the
+`3c` command-line tool, the target is `3c`, but assuming you want to
 actually compile the Checked C source files generated by 3C, you'll
 also want to build `clang` (the Checked C compiler, which is a
 modified version of Clang). Executables will end up in `build/bin`;
-you'll likely want to add this directory to your `$PATH`.
+you'll likely want to add this directory to your `$PATH` or write
+suitable wrapper scripts.
+
+Here is [how to use the `3c` tool](../../../tools/3c/README.md) once
+you build it.
 
 ## Build options
 
@@ -62,6 +75,10 @@ may reduce the build time and/or peak memory use:
 
 - Pass `-DLLVM_OPTIMIZED_TABLEGEN=ON`.
 
+- Pass `-DLLVM_APPEND_VC_REV=OFF` to turn off embedding of your Git
+  head commit ID in the executables and thus avoid the need to re-link
+  all of them every time the commit ID changes.
+
 (The reference for improving build performance is [this LLVM
 page](https://www.llvm.org/docs/GettingStarted.html#common-problems),
 but we have attempted to describe the most promising options here.)
@@ -84,23 +101,3 @@ https://developer.apple.com/download/more/, log in with your Apple
 Developer ID, and download "Command Line Tools for Xcode 11.3.1".
 Then, enable these by invoking (from the command line), `sudo
 xcode-select -r`.
-
-## CR/LF issues
-
-There is a [known
-problem](https://github.com/correctcomputation/checkedc-clang/issues/317)
-where git shows source files in a clean checkout as having modified
-line endings, even if you only ever use Linux. The problem seems to
-occur less often with newer versions of git. We believe the root cause
-is that [the root `.gitattributes` file](../../../../.gitattributes)
-inaccurately states that all files have LF endings, but some have CRLF
-endings in the repository; we will try to fix that when we can. In the
-meantime, running the following commands in a working tree that is
-experiencing the problem seems to make it go away, at least
-temporarily:
-
-```
-git rm .gitattributes
-git status
-git checkout HEAD -- .gitattributes
-```

clang/docs/checkedc/3C/README.md

@@ -23,28 +23,88 @@ some point.
 
 ## What 3C users should know about the development process
 
-Development of 3C is led by [Correct Computation,
-Inc.](https://correctcomputation.com/) (CCI) in the
+Checked C development as a whole is led by Microsoft in the
+https://github.com/microsoft/checkedc-clang repository. 3C is included
+in the Checked C codebase, but 3C development is led by [Correct
+Computation, Inc.](https://correctcomputation.com/) (CCI) in the
 https://github.com/correctcomputation/checkedc-clang repository, which
-is a fork of Microsoft's Checked C repository
-(https://github.com/microsoft/checkedc-clang). Issues and pull
-requests related to 3C should be submitted to the CCI repository; see
-[CONTRIBUTING.md](CONTRIBUTING.md) for more information. Changes are
-merged frequently from Microsoft's repository to CCI's and less
-frequently in the opposite direction. While some automated tests of 3C
-are run in Microsoft's repository, the coverage of these tests is
-currently mediocre, so changes to Microsoft's repository may break
-some functionality of its copy of 3C. Thus, users of 3C can choose
-either CCI's repository (for the latest 3C with a somewhat older
-Checked C) or Microsoft's (for the latest Checked C with 3C that is
-significantly older and possibly broken). This workflow may change in
-the future.
-
-As of November 2020, 3C is pre-alpha quality and we are just starting
+is a fork of Microsoft's repository; changes are periodically merged
+back and forth. (That is, CCI plays roughly the role of a ["subsystem
+maintainer"](https://www.kernel.org/doc/html/latest/process/2.Process.html#how-patches-get-into-the-kernel)
+for 3C, for those familiar with that term from Linux kernel
+development.) Both 3C and the rest of the Checked C tools (including
+the compiler) are available in both repositories, but the repositories
+will generally have different versions of different parts of the code
+at any given time, so [you'll have to decide which repository is more
+suitable for your work](#which-checkedc-clang-repository-to-use).
+
+Issues and pull requests related to 3C should be submitted to CCI's
+repository; see [CONTRIBUTING.md](CONTRIBUTING.md) for more
+information.
+
+As of March 2021, 3C is pre-alpha quality and we are just starting
 to establish its public presence and processes. CCI is also working on
 a proprietary extension of 3C called 5C ("**C**orrect
 **C**omputation's **C**hecked-**C**-**C**onvert"). Our current plan is
 that 3C will contain the core inference logic, while 5C will add
 features to enhance developer productivity. If you'd like more
 information about 5C, please contact us at
 [email protected].
+
+### Which `checkedc-clang` repository to use?
+
+We typically recommend that serious 3C users use CCI's repository to
+get 3C fixes and enhancements sooner, but in some scenarios, you may
+be better off with Microsoft's repository. Here, in detail, are the
+factors we can think of that might affect your decision (as of
+March 2021):
+
+- CCI strives to merge changes reasonably quickly from Microsoft's
+  repository to CCI's, but most 3C-specific changes are made first in
+  CCI's repository and merged to Microsoft's in batches every few
+  months. Thus, CCI's repository typically gives you a significantly
+  newer version of 3C, while Microsoft's repository typically gives
+  you a somewhat newer version of the rest of the Checked C codebase,
+  including some shared code used by 3C. The implication of that last
+  point is that a fix to the shared code in Microsoft's repository
+  would benefit Microsoft's copy of 3C immediately but would not take
+  effect on CCI's copy until the next merge from Microsoft to CCI, so
+  CCI's copy of 3C is not always newer in _all_ respects.
+
+- While the [3C regression tests](CONTRIBUTING.md#testing) are run on
+  every change to either repository, CCI's copy of 3C undergoes
+  additional testing: a set of "benchmark" tests that run nightly,
+  plus manual use by CCI engineers. If a change to CCI's repository
+  passes the regression tests but breaks the additional tests, we will
+  generally fix the problem quickly. But if the same happens in
+  Microsoft's repository, functionality of Microsoft's copy of 3C that
+  is not covered by the regression tests could remain broken for some
+  time. The problem would likely be detected the next time CCI tries
+  to merge from Microsoft, in which case we would try to get a fix
+  into Microsoft's repository reasonably quickly.
+
+- The 3C regression tests are run regularly on Windows on Microsoft's
+  repository but not on CCI's. A change that breaks the regression
+  tests on Windows generally won't be made to Microsoft's repository,
+  but such a change could be made to CCI's and the problem may not be
+  detected until the next time CCI tries to merge to Microsoft (or a
+  user reports the problem to us). So you're less likely to encounter
+  Windows-specific problems with Microsoft's repository.
+
+- On the other hand, some CCI engineers work on Mac OS X and
+  frequently run the regression tests and other manual tests on CCI's
+  copy of 3C on Mac OS X, while we are unaware of any testing of
+  Microsoft's copy of 3C on Mac OS X. So you may be less
+  likely to encounter Mac-specific problems with CCI's repository. But
+  so far, when we've seen Mac-specific problems, we've usually gotten
+  a fix into Microsoft's repository reasonably quickly.
+
+As noted in the [setup instructions](INSTALL.md#basics), both 3C and
+the Checked C compiler depend on the Checked C system headers, which
+Microsoft maintains in [a subdirectory of a separate repository named
+`checkedc`](https://github.com/microsoft/checkedc/tree/master/include).
+CCI has [a fork of this
+repository](https://github.com/correctcomputation/checkedc), but
+currently it is used only for submitting changes to Microsoft. All
+users should use Microsoft's `checkedc` repository regardless of which
+`checkedc-clang` repository they use.