proposal: all: add bare metal support

[abarisani]

Proposal Details

I propose the addition of a new GOOS target, such as GOOS=none, to allow Go runtime execution under specific application defined exit functions, rather than arbitrary OS syscalls, enabling freestanding execution without direct OS support.

This is currently implemented in the GOOS=tamago project, but for reasons laid out in the Proposal Background section it is proposed for upstream inclusion.

Go applications built with GOOS=none would run on bare metal, without any underlying OS. All required support is provided by the Go runtime and external driver packages, also written in Go.

Go runtime changes

Note

The changes are also documented in package tamago/doc

A working example of all proposed changes can be found in the GOOS=tamago implementation.

Board support packages or applications would be required (only under GOOS=none) to define the following functions to support the runtime.

If the use of go:linkname is undesirable different strategies are possible, right now linkname is used as convenient way to have externally defined functions being directly invoked in the runtime early on.

These hooks act as a "Rosetta Stone" for integration of a freestanding Go runtime within an arbitrary environment, whether bare metal or OS supported.

For bare metal examples see the following packages: usbarmory, uefi, microvm.

For OS supported examples see the following tamago packages: linux, applet.

Index

• Variables
• func GetRandomData(b []byte)
• func Hwinit0()
• func Hwinit1()
• func InitRNG()
• func Nanotime() int64
• func Printk(c byte)

Variables

Bloc describes the optional override of runtime.Bloc to redefine the heap memory start address, this is typically only required on OS supported environments.

For an example see package linux.

var Bloc uintptr

Exit describes the optional set of runtime.Exit to define a runtime termination function.

For an example see package microvm.

var Exit func(int32)

Idle describes the optional set of runtime.Idle to define a CPU idle function.

For a basic example see package amd64, a more advanced example involving a physical countdown timer such as [arm.CPU.SetAlarm] is implemented in the tamago example.

var Idle func(until int64)

RamSize, which must be linked as [runtime.ramSize]¹, defines the total size of the physical or virtual memory available to the runtime for allocation (including the code segment which must be mapped within).

For an example see package microvm memory layout.

¹ //go:linkname RamSize runtime.ramSize

var RamSize uint

RamStackOffset, which must be linked as [runtime.ramStackOffset]¹, defines the negative offset from the end of the available memory for stack allocation.

For an example see package amd64.

¹ //go:linkname RamStackOffset runtime.ramStackOffset

var RamStackOffset uint

RamStart, which must be linked as [runtime.ramStart]¹, defines the start address of the physical or virtual memory available to the runtime for allocation (including the code segment which must be mapped within).

For an example see package amd64 memory layout.

¹ //go:linkname RamStart runtime.ramStart

var RamStart uint

SocketFunc describes the optional override of net.SocketFunc to provide the network socket implementation. The returned interface must match the requested socket and be either net.Conn, net.PacketConn or net.Listen.

For an example see package vnet.

var SocketFunc func(ctx context.Context, net string, family, sotype int, laddr, raddr Addr) (interface{}, error)

func GetRandomData

func GetRandomData(b []byte)

GetRandomData, which must be linked as [runtime.GetRandomData]¹, generates len(b) random bytes and writes them into b.

For an example see package amd64 random number generation.

¹ //go:linkname GetRandomData runtime.getRandomData

func Hwinit0

func Hwinit0()

Hwinit0, which must be linked as [runtime.hwinit0]¹, takes care of the lower level initialization triggered before runtime setup (pre World start).

It must be defined using Go's Assembler to retain Go's commitment to backward compatibility, otherwise extreme care must be taken as the lack of World start does not allow memory allocation.

For an example see package amd64 initialization.

¹ //go:linkname Hwinit0 runtime.hwinit0

func Hwinit1

func Hwinit1()

Hwinit1, which must be linked as [runtime.hwinit1]¹, takes care of the lower level initialization triggered early in runtime setup (post World start).

For an example see package microvm platform initialization.

¹ //go:linkname Hwinit1 runtime.hwinit1

func InitRNG

func InitRNG()

InitRNG, which must be linked as [runtime.initRNG]¹, initializes random number generation.

For an example see package amd64 randon number generation.

¹ //go:linkname InitRNG runtime.initRNG

func Nanotime

func Nanotime() int64

Nanotime, which must be linked as [runtime.nanotime1]¹, returns the system time in nanoseconds.

It must be defined using Go's Assembler to retain Go's commitment to backward compatibility, otherwise extreme care must be taken as the lack of World start does not allow memory allocation.

For an example see package fu540 initialization.

¹ //go:linkname Nanotime runtime.nanotime1

func Printk

func Printk(c byte)

Printk, which must be linked as [runtime.printk]¹, handles character printing to standard output.

It must be defined using Go's Assembler to retain Go's commitment to backward compatibility, otherwise extreme care must be taken as the lack of World start does not allow memory allocation.

For an example see package usbarmory console handling.

Interrupt handling helpers

The Go runtime would implement the following, or similar, functions to aid interrupt handling:

• runtime.GetG (example), runtime.WakeG (example), runtime.Wake (example): asynchronous goroutine wake-up

// GetG returns the pointer to the current G and its P.
func GetG() (gp uint64, pp uint64)

// WakeG modifies a goroutine cached timer for time.Sleep (g.timer) to fire as
// soon as possible.
//
// The function is meant to be invoked within Go assembly and its arguments
// must be passed through registers rather than on the frame pointer, see
// definition in sys_tamago_$GOARCH.s for details.
func WakeG()

// Wake modifies a goroutine cached timer for time.Sleep (g.timer) to fire as
// soon as possible.
func Wake(gp uint)

Compilation

The compilation of such targets would remain identical to standard Go binaries, while the loading strategy might differ depending on the hardware but anyway be handled in a manner completely external to the Go distribution, using standard flags as required, examples:

# Example for Cloud Hypervisory, QEMU and Firecracker KVMs
GOOS=tamago GOARCH=amd64 ${TAMAGO} build -ldflags "-T 0x10010000 -R 0x1000" main.go

# Example for USB armory Mk II
GOOS=tamago GOARM=7 GOARCH=arm ${TAMAGO} build -ldflags "-T 0x80010000 -R 0x1000" main.go

# Example for QEMU RISC-V sifive_u
GOOS=tamago GOARCH=riscv64 ${TAMAGO} build -ldflags "-T 0x80010000 -R 0x1000" main.go

# Example for Linux userspace
GOOS=tamago ${TAMAGO} build main.go

Proposal Background

This proposal follows updates on the TamaGo project, which brings bare metal execution for Go on AMD64, ARM and RISCV64 targets.

While similar proposals (see #37503 and #46802) have been already attempted without success, this last effort is motivated by considerable advancements and changes in our effort.

Notable changes:

1. There is now fully tested Go standard library support, integrated with vanilla distribution tests. The testing environment for AMD64, ARM and RISCV64 architectures runs under Linux natively or using qemu-user-static via binfmt_misc.

2. The tamago networking code allows external definition of a single socket function to attach gVisor or any other fake networking stack if desired, this could benefit other Go architectures and allow replacement of existing fake networking for js/wasip1.

3. Because of what implemented to support 1. and 2. GOOS=tamago allows the execution of unmodified Go applications as softly isolated userspace code with OS resources, such as networking and filesystems, isolated from the actual user OS.

4. TamaGo now no longer focuses only on ARM embedded systems, but it extends to AMD64 KVM execution such as microVMs.

5. The overall Go distribution changes are free from hardware dependent code (e.g. peripheral drivers) and changes are unified across different architectures, see for instance the identical implementation for entry points of amd64, arm, riscv64 architectures.

In summary GOOS=tamago has been transformed to a generic implementation that allows execution without relying on operating system calls but rather with a unified "outside world" exit interface, whether implemented for testing, userspace execution, real hardware or paravirtualization.

On ARM we implemented bootloaders, Trusted Execution Environments and full secure OS and applets with this framework.

Thanks to a recent amd64 port, we enabled Pure Go KVMs under Cloud Hypervisor, Firecracker and QEMU.

This also allowed us to implemented execution under UEFI which enabled 100% Go EFI applications and bootloaders such as go-boot (which currently booted Linux on the Thinkpad I am writing from).

We think adopting these compact Go distribution changes would allow not only preservation of this ecosystem but expansion and innovation of the Go language as a whole, taking it to surprising, unxpected, yet ideal, environments.

The cost of maintaining the patch is, in our opinion, reasonable and if anything beneficial in improving the Go abstraction across architectures and OS specific components.

The only component that is somewhat low-level and sensitive in terms of maintenance between major Go releases is our asynchronous goroutine waking function, which is used to serve external interrupt requests.

However such function can be re-implemented in a manner consistent with existing OS signaling or, simply with the awareness and inclusion of GOOS=none, hooked to a much simpler standardized interface within the timer
structure, which so far has not been implemented purely to avoid any pollution with non-tamago architectures.

[gabyhelp]

Related Issues

• proposal: all: add bare metal ARM support #46802 (closed)
• proposal: all: add bare metal ARM support #37503 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

Looking at the past proposals, this is just a list of new changes (much like the previous), but doesn't actually address the primary concerns of the past proposals, namely the shift of support burden on to the core team.

#35956 (comment)
#37503 (comment)
#46802 (comment)

[abarisani]

Looking at the past proposals, this is just a list of new changes (much like the previous), but doesn't actually address the primary concerns of the past proposals, namely the shift of support burden on to the core team.

#35956 (comment) #37503 (comment) #46802 (comment)

You are right, I just separated an actual proposal with the underlying background, hopefully this should reason the new request.

I've laid out the external functions proposal which are specifically proposed to shift the support burden of any hardware or target specific support away from the core team.

[FiloSottile]

I would like us to take another look at this. I think there are at least two major changes which affect the maintenance burden vs. value tradeoff evaluation:

• First, GOOS=none and the OS interaction interface make this much more useful, as it's not just a way to run Go on tiny ARM boards, but lets applications implement support for disparate environments, including hypervisors and pre-OS environments.

• Second, it's been five years since the first proposal and four years since the last one. @abarisani's team has been maintaining the fork for years now, which lends credibility to their commitment to maintain the port in tree, as well. (In particular, I see they implemented support for running the standard library tests on a regular Linux OS, which is something I had suggested would be necessary to carry a port with a good builder.)

This span of time also makes it possible to have a retrospective look at use cases, and at the actual burden/churn.

I probably over-index on security-sensitive use cases for obvious reasons, but running Go applications sandboxed as Firecracker micro VMs and writing memory-safe UEFI bootloader firmware in Go is extremely exciting to me.

[seankhliao]

What about the other parts of the porting policy: https://go.dev/wiki/PortingPolicy#requirements-for-a-new-port

Besides @abarisani , who are the named maintainers going to be, and what level of user adoption have we seen with tamago over the years?

[prattmic]

cc @golang/runtime

[prattmic]

I am also excited by the idea of bare metal Go.

To me, the major challenges are around compatibility. Presumably such a port should conform to the Go 1 Compatibility Promise. This has two major components:

1. A stable API for the runtime's dependencies, which will be supported ~indefinitely. Your proposal lays out the current tamago runtime API, and while I agree it needs a better mechanism than linkname, I don't think it is too bad. I suspect we could come up with something we're willing to commit to here. As an aside, I have heard similar requests for a runtime API from folks who would like to run Go on minimal WASM runtimes that don't necessarily support WASI. Presumably this API would work for GOOS=none GOARCH=wasm as well.

2. A stable subset of the language that can be used to implement the runtime dependency APIs. This is the really scary part to me. "runtime Go" is already a specialized way of writing Go, which is mostly unspecified and unstable. But it is in tree. If a changed compiler heuristic adds an allocation somewhere in the runtime that can't allocate, that's OK, we can change the runtime. With GOOS=none that would be an incompatible change. So I think we'd need to define and support a simple subset of the language that is guaranteed safe to use for implementing these APIs, which does not seem like a small project.

Alternatively, we could punt on this and state that these APIs must not be implemented in Go at all, and instead must be implemented in assembly. That seems unfortunate, though looking at your example, the code is fairly simple so maybe this wouldn't be completely untenable.

[prattmic]

I see Tamago is currently single-threaded (according to the lock implementation in src/runtime/lock_tamago.go). Presumably the virtualized unikernel use case will want multi-threading. Is this something on your radar, and do you have a sense of how it would affect the runtime API?

[abarisani]

What about the other parts of the porting policy: https://go.dev/wiki/PortingPolicy#requirements-for-a-new-port

Besides @abarisani , who are the named maintainers going to be, and what level of user adoption have we seen with tamago over the years?

We are happy to provide two developers and accept responsibility as well as maintaining a builder, we are happy to provide any support needed and required.

Our existing effort has been engineered specifically to avoid touching first class ports or change them in any way, the implementation of standard distribution tests has also been specifically worked to support such requirements and a builder.

Concerning adoption we have WithSecure sponsored projects as well as Transparency.dev Armory Witness projects (bootloader, OS, applet) as primary adopters of tamago (see a list here). We have several thousands units running tamago on a variety of privately contracted projects that unfortunately we cannot speak publicly about. The amd64 port, which enables non-embedded use cases, however is very recent has it has been published only a few months ago, we do think that a side effect of upstream adoption would also be increase in popularity of Go in places people generally don't see a use for (go-boot being one example).