self-hosted compiler: ship it!

[andrewrk]

This is a tracking issue for when we can completely replace stage1 with self-hosted in official builds of Zig.

• add debug info to the LLVM backend
  • initial implementation stage2 LLVM debug info #11085
  • implement debug info for structs in the llvm backend #11095
  • implement debug info for unions in the llvm backend #11096
  • implement debug info for global variables in the llvm backend #11097
  • stage2: add debug info for locals in the LLVM backend #11145
  • make function calls contain debug stmt line/column info #11158
  • make else if conditions contain debug stmt info #11160
• all runtime safety tests passing for stage2 #11674
• get compilation errors into an acceptable state
  • proposal: improved compiler test suite (compiler errors, run and compare, translate-c, etc.) #11288
  • missing compile error for runtime iteration over compile-time-only array elements #11632
• adjust the default backend to be LLVM until other backends reach full behavior test coverage
• async/await/suspend/resume #6025 async/await/suspend/resume will not be available in 0.10.0
• Audit and consolidate stage1/stage2 tests. All behavior tests passing for self-hosted. #6034
• stage2: array-like tuple initialization gets LLVM verification panic #11159
• std.fmt.parseFloat does not parse 9007199254740993.0 correctly #11169
• stage2: Sema cannot distinguish vectors from arrays in expressions #11251
• implement error return tracing in the self-hosted compiler #11259
• all std lib tests passing with the LLVM backend
  • Stage2: simple if expression randomly has wrong result when safety checks are enabled #11898
  • Stage2 miscompiles call to std.math.big.int.Managed.shiftLeft #11897
  • Stage2 specific union segfaults test runner when used in error union or optional #11857
  • Stage2 miscompiles vector element acces via slice of vectors #11856
  • Stage2 can't negate a NaN at runtime #11853
  • struct with field default initialized to size of the struct gives false positive 'depends on itself' error #12029
• stage3 fails the behavior tests #11450
• stage2-generated binaries are slower and more bloated than stage1-generated binaries #11498
• implement noinline #11767
• Upgrade to LLVM 14 #11274
• full test suite passing with the LLVM backend
• check all the TODO comments in stage2 source code and convert relevant ones into issues blocking this one convert TODO comments from source code into issues #363
• Eliminate use of conditional compilation on zig_backend in compiler-rt, freestanding libc, test runner, and default panic
• use the real start code
• eliminate conditional compilation on zig_backend in the std lib
• Eliminate the other TODO items from generated builtin.zig file
• work with individual third party projects to make sure the stage2 transition completes successfully
  • Bun (cc @Jarred-Sumner)
  • zig-gamedev (cc @michal-z)
  • mach (cc @slimsag)
  • River (cc @ifreund)
  • aquila (cc @nektro)
  • https://github.com/ominitay/zigvale-barebones (cc @ominitay)
  • ncdu
  • https://github.com/ZigEmbeddedGroup/microzig (zig build -Drelease-small)
  • https://github.com/mattnite/gyro (zig build, good amount of zig and C code in here)
  • ZLS
  • https://github.com/sin-ack/zigself
  • https://github.com/floooh/sokol-zig

[andrewrk]

When we release zig 1.0.0, we'll actually have the self hosted compiler built and ready to go, passing all tests. Both projects will be maintained until 2.0.0.

[andrewrk]

There are some compelling reasons to self-host:

• Makes f128 work without needing gnu extensions
• We can write the doc generator in userland
• Makes builtin overflow stuff work without C compiler extensions
• Ability to easily cross-compile the compiler

We still want the bootstrapping process to be simple though. So here's another proposal. We get a self-hosted compiler going right now. It's the official zig compiler. However the C++ implementation must be able to build the official zig compiler. As long as that remains true, bootstrapping is 1 step process.

[andrewrk]

Things I Want to Improve in the Self-Hosted Compiler

Performance and Caching

• Max out performance of machine with thread pool and async I/O
• Pipeline all the work. Split the job up into individual functions that each produce .o files. We'll have LLVM spitting out .o files before the last source file has been tokenized.
• No mutexes. When a coroutine needs a resource that another thread is working on, it yields to another job, getting resumed (in userspace!) when the resource is available (this looks like async/await)
• Multi-layer caching. Cache files, cache AST, cache individual functions
• Establish a file system watch on source files, detecting changes, running through the pipeline (taking caching into account), and atomically update output files in place. The compiler is a long lived process and some of the caching happens in memory.
• Handle temporary out of memory situations with emitting an event that says "waiting for more memory to be available" and it prints how much was needed along with how much the system has available

Representation of Types and Values

ConstExprValue right now has a lot of footguns built into it,
and it wastes memory. The new data layout should accomplish
these things:

• Use a minimal amount of memory
• Have at the very least runtime safety for wrong union field access
  and hopefully more compile errors when adding and removing fields.
• In Stage 1, the Type tells how to interpret the Value. In self-hosted,
  we should divorce these concepts. This should make comptime casting
  more correctly represented.
• Introduce lazy values. For example, if you do @sizeOf(error), this
  can create Value that is backed by a LazyComptimeExpr. We could still
  find out @typeOf(@sizeOf(error)) without causing the lazy expr to
  evaluate. Once we get to the end of the compilation, we start evaluating
  all the lazy expressions. If a lazy expression depends on another lazy
  expression, it gets skipped, and we make a note to start over once done.
  If all lazy expressions must be skipped, then it's a compile error, and
  we show the dependency loop.

[phase]

We'll have LLVM spitting out .o files before the last source file has been tokenized.

How will this work with name resolution? You can't compile a file that depends on a file that hasn't been parsed yet.