proposal: runtime/cgo: move built-in C helpers from cmd/cgo to runtime/cgo

[qmuntal]

cmd/cgo generates on the fly several Go functions, namely C.GoString, C.GoStringN, C.GoBytes, C.CString, and C.CBytes.

These functions inherit most of cgo usability problems despite being plain Go functions, most notably they do not play well with autocompletion. Additionally, its difficult to learn how to use them, and even know they exist, as they are only documented at https://pkg.go.dev/cmd/cgo in the middle of a long guide about how to use cgo, when one would expect them to appear in the Functions section.

I propose that we stop treating these function in a special way by defining and implementing them in runtime/cgo.

In order to maintain backwards compatibility, the autogenerated C functions won't be removed but they will just be a wrapper for the new runtime/cgo functions.

The proposed API is almost the same as the one already generated by runtime/cgo, with the only difference that GoStringN and GoBytes accept an int instead of a C.int as length parameter.

// GoString converts the C string p into a Go string.
func GoString(p *C.char) string

// GoStringN converts the C data p with explicit length to a Go string.
func GoStringN(p *C.char, l int) string

// GoBytes converts the C data p with explicit length to a Go []byte.
func GoBytes(p unsafe.Pointer, l int) []byte 

// CString converts the Go string s to a C string.
//
// The C string is allocated in the C heap using malloc.
// It is the caller's responsibility to arrange for it to be
// freed, such as by calling C.free (be sure to include stdlib.h
// if C.free is needed).
func CString(s string) *C.char

// CBytes converts the Go []byte slice b to a C array.
//
// The C array is allocated in the C heap using malloc.
// It is the caller's responsibility to arrange for it to be
// freed, such as by calling C.free (be sure to include stdlib.h
// if C.free is needed).
func CBytes(b []byte) unsafe.Pointer

[ericlagergren]

I like this idea. However, I believe it will require some sort of compiler magic since C.* types are different across package boundaries.

$ cat main.go
package main

/*
static inline const char* foo() {
	return "hello from C";
}
*/
import "C"

import "example.com/ctest/b"

func main() {
	p := C.foo()
	s := b.GoString(p)
	println(s)
}
$ cat b/b.go
package b

import "C"

func GoString(p *C.char) string {
	return "wat"
}
$ go run main.go
# command-line-arguments
./main.go:14:18: cannot use p (variable of type *_Ctype_char) as type *b._Ctype_char in argument to b.GoString

[rsc]

Yes, we could put each name in both C.Foo and cgo.Foo, with cgo translating C.Foo to cgo.Foo.
But why?
Then we will have two names for each of these functions, and people will get told "don't use the C.Foo names anymore, they are deprecated" (even if they are not). And generally it will cause confusion.
Is there a reason in favor of moving them?

I am not worried about having to look in the cmd/cgo docs. There are many cgo details there. Might as well have them all together.

As for tab completion, it seems like gopls could be updated to take care of that issue without putting users through a shift in how they write cgo code. Better to adapt the tools to reality at this point.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[qmuntal]

After reading

I like this idea. However, I believe it will require some sort of compiler magic since C.* types are different across package boundaries. @ericlagergren

That's a very good point, before posting this proposal I did a quick prototype to verify it was feasible, but did not validate calling cgo.CString from another package. Digging a little bit in the issues backlog I see it is related to the long-standing #13467. Until that is resolved there is no way to treat the proposed API as normal Go functions (aka no compiler trick involved), as they contain C types on its signature. We could also reformulate the API to accept unsafe.Pointer instead of *C.char, but that would be awkward.

Then we will have two names for each of these functions, and people will get told "don't use the C.Foo names anymore, they are deprecated" (even if they are not). And generally it will cause confusion.
Is there a reason in favor of moving them? @rsc

These are the features that are missing:

• Discoverability: These functions do not exist until they are used, the Go code compiles and the CGO definitions can be generated. Moreover, it is not possible to reuse definitions from other packages due to the way cgo generates functions. This is hard to fit in current autocompletion tools without special-casing.

• Referenceability (is this a word?): There is no way to link to the definition of these functions, i.e. one can navigate to io.Writer but not to https://pkg.go.dev/cmd/cgo#CString. This makes it hard for humans to link these functions in discussions or code reviews, and for tools in docs popups.

Additionally, and probably nitpicking, having Go functions (which accepts Go types) exposed in the C package goes against the principle of least surprise, as I would expect to find in that package pure C symbols.

Better to adapt the tools to reality at this point. @rsc

It's true that this ship might already sailed. Improving the tools and the documentation for these functions will also solve most of my issues.

I will file a proposal to special-case these functions in gopls and also submit some commits that I think will improve their documentation, p.e. instructing cmd/cgo to add meaningful comments to the autogenerated functions so it can be displayed by current tooling.

[gopherbot]

Change https://go.dev/cl/386414 mentions this issue: cmd/go: add comments to C.* special functions

[rsc]

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

[rsc]

No change in consensus, so declined.
— rsc for the proposal review group