reflect: documentation about "Conversion of a reflect.SliceHeader or reflect.StringHeader Data field to or from Pointer" is misleading #41705

GingerMoon · 2020-09-30T06:20:18Z

The document about "Conversion of a reflect.SliceHeader or reflect.StringHeader Data field to or from Pointer" is misleading.

var s string
hdr := (*reflect.StringHeader)(unsafe.Pointer(&s)) // case 1
hdr.Data = uintptr(unsafe.Pointer(p))              // case 6 (this case)
hdr.Len = n

It would be great if the doc specifies one more time the risk of the code below:

hdr.Data = uintptr(unsafe.Pointer(p))

When p point to a temporary memory, we will have problem. For example,

func demo(val_copy int64) *reflect.StringHeader {
	var s string
	hdr := (*reflect.StringHeader)(unsafe.Pointer(&s)) 			// case 1
	hdr.Data = uintptr(unsafe.Pointer(&val_copy))               // case 6 (this case)
	hdr.Len = 8
	return hdr
}

func main() {
	val := int64(0xab)
	hdr := demo(val)
	s := *(*string)(unsafe.Pointer(hdr))
	fmt.Printf("%x", s)
}

The tricky thing is, it's often that the output is the same as expectation.
But this is a bug because &val_copy is not referenced by others except uintptr, hence the content of the memory pointed by &val_copy is undefined after the demo function returned.

The text was updated successfully, but these errors were encountered:

bcmills · 2020-09-30T13:43:26Z

this is a bug because &val_copy is not referenced by others except uintptr, hence the content of the memory pointed by &val_copy is undefined after the demo function returned.

I don't think that actually is a bug: assigning &val_copy to hdr.Data causes it to be referenced by the variable s, which is itself referenced via the returned *reflect.StringHeader. So the compiler should report that val_copy escapes, and therefore allocate it on the heap.

And indeed it does exactly that (https://play.golang.org/p/nYH-4TVrXWU):

example.com$ go build -gcflags=-m -o=/dev/null .
# example.com
./main.go:9:6: can inline demo
./main.go:19:13: inlining call to demo
./main.go:21:12: inlining call to fmt.Printf
./main.go:9:11: moved to heap: val_copy
./main.go:10:6: moved to heap: s
./main.go:19:13: moved to heap: val_copy
./main.go:21:13: s escapes to heap
./main.go:21:12: []interface {}{...} does not escape
<autogenerated>:1: .this does not escape

CC @mdempsky

bcmills · 2020-09-30T13:45:08Z

That said, it would be much clearer for the demo function to return the string rather than the *reflect.SliceHeader: https://play.golang.org/p/HD-XVm2iSZi

func demo(val_copy int64) string {
	var s string
	…
	return s
}

But that's a matter of readability, not correctness.

bcmills · 2020-09-30T13:48:26Z

@GingerMoon, given the above, is there some other way you think we should clarify the documentation?

mdempsky · 2020-09-30T19:16:20Z

@GingerMoon As I understand your concern, it's that: (1) the documentation suggests your sample program will always print "ab00000000000000" (at least on little-endian CPUs), but (2) the Go compilers do not guarantee this. Do I understand that concern correctly?

If so, then concern 2 is mistaken. The Go compilers do guarantee that output, and the documentation is correct to suggest it.

However, I'm happy to understand the source of misunderstanding here so that we can improve the documentation and prevent future misunderstandings.

GingerMoon · 2020-09-30T22:55:52Z

Thanks you guys. Before talking about the document, I would like to share a strange case to show you why I have a higher expectation on the document.
(Sorry I cannot access the playground.)

package main

import (
	"bytes"
	"fmt"
	reflect "reflect"
	"sync"
	"unsafe"
)

var (
	n = 1_000_000
)

func demo(val int64) []byte {
	hdr := reflect.SliceHeader{Data: uintptr(unsafe.Pointer(&val)), Len: 8, Cap: 8}
	return *(*[]byte)(unsafe.Pointer(&hdr))
}

func test() {
	val := int64(0xab)

	ch := make(chan []byte, 1_000_000)
	go func() {
		expectation := []byte{0xab,0,0,0, 0,0,0,0}
		for r := range ch {
			if bytes.Compare(expectation, r) != 0 {
				fmt.Printf("expectation: %x, but result is: %x", expectation, r)
				panic("oops")
			}
		}
	}()

	wg := &sync.WaitGroup{}
	wg.Add(n)
	for i := 0; i < n; i++ {
		go func() {
			defer wg.Done()
			r := demo(val)
			ch <- r
		}()
	}
	wg.Wait()
	close(ch)
	fmt.Println("done")
}
func main() {
	for {
		test()
	}
}

The result returned by demo is not ALWAYS the same as our expectation.
The root cause is, hdr.Data references &val_copy via uintptr.
But uintptr is not a "strong" reference to &val_copy.
So the memory pointed by &val_copy is invalid after the function returns.
Please correct me if my understanding is wrong. Any insight would be much appreciated.

/ Even if a uintptr holds the address of some object,
// the garbage collector will not update that uintptr's value
// if the object moves, nor will that uintptr keep the object
// from being reclaimed.

//  // INVALID: a directly-declared header will not hold Data as a reference.
// var hdr reflect.StringHeader
// hdr.Data = uintptr(unsafe.Pointer(p))
// hdr.Len = n
// s := *(*string)(unsafe.Pointer(&hdr)) // p possibly already lost

mdempsky · 2020-09-30T23:05:30Z

The issue with that code is you've declared a variable hdr of type reflect.SliceHeader, and then tried converting it to []byte using *(*[]T)(unsafe.Pointer(&hdr)). That code is not supported. (And cmd/vet should warn about it in Go 1.16: #40701.)

The only supported unsafe use of reflect.SliceHeader is to use *reflect.SliceHeader as a pointer to an actual slice type. The Go compiler has code that specially handles *reflect.SliceHeader, but not reflect.SliceHeader.

GingerMoon · 2020-09-30T23:41:29Z

The demo func has two problems:

1. Incorrect usage of SliceHeader , the document is clear for this.
1. Reference the memory pointed by &val_copy by uintptr. uintptr doesn't make val_copy escape to the heap.
  After demo returns, the content of the memory pointed by &val_copy becomes invalid.

This bug is difficult to debug becasue most of the time, the demo returns the expected result.

The sample in the doc below makes people think that hdr.Data = uintptr(unsafe.Pointer(p)) is a correct usage without thinking of the restrictation of using uintptr.
Especially there is a case 6 comment which makes peopole think they are using uintptr correctly.

var s string
hdr := (*reflect.StringHeader)(unsafe.Pointer(&s)) // case 1
hdr.Data = uintptr(unsafe.Pointer(p))              // case 6 (this case)
hdr.Len = n

So I'd like to change the document to remind people the risk of using uintptr.
Also add some more info (not just gc) to the doc below.

// Even if a uintptr holds the address of some object,
// the garbage collector will not update that uintptr's value
// if the object moves, nor will that uintptr keep the object
// from being reclaimed.

Please correct me if my understanding is wrong.

mdempsky · 2020-09-30T23:57:50Z

Reference the memory pointed by &val_copy by uintptr. uintptr doesn't make val_copy escape to the heap.
After demo returns, the content of the memory pointed by &val_copy becomes invalid.

This is only an issue because of the first issue. The rules allow converting pointers to uintptr for the purposes of storing into the Data field of a *reflect.SliceHeader or *reflect.StringHeader. When unsafe.Pointer is used correctly, the compiler knows val_copy escapes to the heap.

For example, this code that you originally shared is allowed:

func demo(val_copy int64) *reflect.StringHeader {
	var s string
	hdr := (*reflect.StringHeader)(unsafe.Pointer(&s)) 			// case 1
	hdr.Data = uintptr(unsafe.Pointer(&val_copy))               // case 6 (this case)
	hdr.Len = 8
	return hdr
}

GingerMoon · 2020-10-01T01:10:54Z

Thanks to @mdempsky for sharing the insight!
How about appending this insight into the document sample code?

//	var s string
//	hdr := (*reflect.StringHeader)(unsafe.Pointer(&s)) // case 1
//	hdr.Data = uintptr(unsafe.Pointer(p))              // case 6 (this case). 
//	hdr.Len = n
//
// In this usage hdr.Data is really an alternate way to refer to the underlying
// pointer in the string header, not a uintptr variable itself.

// However, in the INVALID case below, hdr.Datar will not be the "strong" reference to p,
// so p is possible to be lost.

//
// In general, reflect.SliceHeader and reflect.StringHeader should be used
// only as *reflect.SliceHeader and *reflect.StringHeader pointing at actual
// slices or strings, never as plain structs.
// A program should not declare or allocate variables of these struct types.
//
//	// INVALID: a directly-declared header will not hold Data as a reference.
//	var hdr reflect.StringHeader
//	hdr.Data = uintptr(unsafe.Pointer(p))
//	hdr.Len = n
//	s := *(*string)(unsafe.Pointer(&hdr)) // p possibly already lost

ianlancetaylor · 2020-10-01T02:42:05Z

It seems to me that your new text basically says the same thing as the INVALID comment in the example.

GingerMoon · 2020-10-01T03:17:27Z

thank you guys for sharing.

GingerMoon · 2020-10-09T07:29:05Z

@ianlancetaylor @mdempsky
I checked the doc again, the doc mentions:

A uintptr is an integer, not a reference. Converting a Pointer to a uintptr creates an integer value with no pointer semantics. Even if a uintptr holds the address of some object, the garbage collector will not update that uintptr's value if the object moves, nor will that uintptr keep the object from being reclaimed.

According to this description about uintptr, the demo below

func demo(val_copy int64) *reflect.StringHeader {
	var s string
	hdr := (*reflect.StringHeader)(unsafe.Pointer(&s)) 			// case 1
	hdr.Data = uintptr(unsafe.Pointer(&val_copy))               // case 6 (this case)
	hdr.Len = 8
	return hdr
}

&val_copy is lost because it's only referenced by hdr.Data, which is uintptr.

In order to show the information below,

The rules allow converting pointers to uintptr for the purposes of storing into the Data field of a *reflect.SliceHeader or *reflect.StringHeader. When unsafe.Pointer is used correctly, the compiler knows val_copy escapes to the heap.

How about refine the doc as below?

A uintptr is an integer, not a reference. Converting a Pointer to a uintptr creates an integer value with no pointer semantics. Even if a uintptr holds the address of some object, the garbage collector will not update that uintptr's value if the object moves, nor will that uintptr keep the object from being reclaimed.
The only exception is when converting pointers to uintptr for the purposes of storing into the Data field of a *reflect.SliceHeader or *reflect.StringHeader.
When unsafe.Pointer is used correctly, the Data field of a *reflect.SliceHeader or *reflect.StringHeader will prevent the object from being lost.
....
var s string
hdr := (*reflect.StringHeader)(unsafe.Pointer(&s)) // case 1
hdr.Data = uintptr(unsafe.Pointer(p)) // case 6 (this case), hdr.Data prevents p from being lost
hdr.Len = n

ianlancetaylor · 2020-10-09T17:45:05Z

I don't think we should make the promise you are describing, even if that is how the compiler works today.

bcmills · 2020-10-09T18:01:50Z

When unsafe.Pointer is used correctly, the Data field of a *reflect.SliceHeader or *reflect.StringHeader will prevent the object from being lost.

It is not the Data field of the SliceHeader or StringHeader that prevents the object from being lost: it is the underlying string or []T variable itself.

The uintptr Data field does not matter, except to the extent that the compiler needs to insert a write barrier if the underlying slot was actually allocated as a pointer. (I suspect that there is a way to generalize that rule beyond reflect.StringHeader and reflect.SliceHeader, but I don't think that the compiler implements the fully-general version of it today.)

GingerMoon · 2020-10-13T08:46:57Z

The document says:

Conversion of a uintptr back to Pointer is not valid in general.

A uintptr is an integer, not a reference. Converting a Pointer to a uintptr creates an integer value with no pointer semantics. Even if a uintptr holds the address of some object, the garbage collector will not update that uintptr's value if the object moves, nor will that uintptr keep the object from being reclaimed.

The remaining patterns enumerate the only valid conversions from uintptr to Pointer.

What if the ojbect pointed by the uintptr is moved in the case below?
I know it is the special case mentioned in the above doc, but I can't see why it's special.

(3) Conversion of a Pointer to a uintptr and back, with arithmetic.

If p points into an allocated object, it can be advanced through the object by conversion to uintptr, addition of an offset, and conversion back to Pointer.

p = unsafe.Pointer(uintptr(p) + offset)

ianlancetaylor · 2020-10-13T19:09:21Z

What if the ojbect pointed by the uintptr is moved in the case below?

Moved when, exactly?

The doc is promising that if you write

p = unsafe.Pointer(uintptr(p) + offset)

then the value of p will continue to point to the same object that it pointed to before. That is guaranteed to be true whether the object moves or not.

GingerMoon · 2020-10-13T23:01:59Z

What if the ojbect pointed by the uintptr is moved in the case below?

Moved when, exactly?

The doc is promising that if you write
p = unsafe.Pointer(uintptr(p) + offset)
then the value of p will continue to point to the same object that it pointed to before. That is guaranteed to be true whether the object moves or not.

The doc mentions the garbage collector can move objects, so "Moved when, exactly?" maybe during evacuation gc phace? Or just one of the gc phases. By the way, does golang gc move objects like Java G1GC? I would appreciate if you could share some links specifying how golang gc works.

Considering the following case:

type S struct {
	a int64
	b int64
}

s := S{1,2}
p := &s

Let's assume p's value 0xaddress1.
After "p = unsafe.Pointer(uintptr(p) + offset)" is executed, p's value 0xaddress1+offset. But later on the original s is moved by gc,
s's address is changed to address2.
But now p's value is still 0xaddress1+offset.

What does golang do in such cases?

The remaining patterns enumerate the only valid conversions from uintptr to Pointer.

Could you please tell me the difference between the valid and invalid conversions?

ianlancetaylor · 2020-10-13T23:14:07Z

The doc mentions the garbage collector can move objects, so "Moved when, exactly?" maybe during evacuation gc phace? Or just one of the gc phases. By the way, does golang gc move objects like Java G1GC? I would appreciate if you could share some links specifying how golang gc works.

In all of the current implementation of Go that I am aware of, the garbage collector never moves objects. However, the stack copier does move objects.

There are various documents that describe different aspects of the garbage collector used by the gc runtime. The basic framework is described at https://golang.org/s/go14gc.

Let's assume p's value 0xaddress1.
After "p = unsafe.Pointer(uintptr(p) + offset)" is executed, p's value 0xaddress1+offset. But later on the original s is moved by gc,
s's address is changed to address2.
But now p's value is still 0xaddress1+offset.

That turns out not to be the case. Go supports pointers to the interior of objects, and when those objects move the interior pointers are updated. If the address of s changes to 0xaddress2, then p will be updated to 0xaddress2 + offset.

Note that Java acts differently, as Java does not support pointers to the interior of objects. But in Go you can get such a pointer by writing something like &s[2] to get the address of an element in an array. If the array moves, that pointer must be updated.

Could you please tell me the difference between the valid and invalid conversions?

The valid ones are guaranteed to work. The invalid ones are not guaranteed to work. I don't know how else to explain it.

GingerMoon · 2020-10-14T11:02:36Z

Thanks a lot @ianlancetaylor for sharing!

Would you please share more about the "stack copier" and when an object can be moved?
Currently my team is using the following ways to convert int64 to a byte slice (currently we don't want to use binary.XXXEndian):

func demo(val *int64) []byte {
	hdr := reflect.SliceHeader{Data: uintptr(unsafe.Pointer(val)), Len: 8, Cap: 8}
	return *(*[]byte)(unsafe.Pointer(&hdr))
}

According to to document

A program should not declare or allocate variables of these struct types.

, so this is an invalid case.

Actually SliceHeader should not be used for this purpose, no matter how we use it.
For example, the following use case are not just "unsafe", they are totally wrong:

func UnsafeCastInt64ToBytes(val *int64) []byte {
    hdr := reflect.SliceHeader{Data: uintptr(unsafe.Pointer(val)), Len: 8, Cap: 8}
    return *(*[]byte)(unsafe.Pointer(&hdr))
}

func UnsafeCastInt64ToBytes2(val *int64) []byte {
    var result []byte
    hdr := (*reflect.SliceHeader)(unsafe.Pointer(&result))
    hdr.Data = uintptr(unsafe.Pointer(val))
    hdr.Len = 8
    hdr.Cap = 8
    return *(*[]byte)(unsafe.Pointer(&hdr))
}

Am I right?
Any insight would be much appreciated!

bcmills · 2020-10-14T13:13:29Z

@GingerMoon, your UnsafeCastInt64ToBytes2 is almost correct — you just need to change the return statement to return result instead of converting the header value itself back to a slice. (My unsafeslice.SetAt takes a similar approach.)

Note that proposal #19367 would make that sort of conversion much simpler in most cases.

randall77 · 2020-10-14T14:51:44Z

currently we don't want to use binary.XXXEndian

Can you explain why?

func UnsafeCastInt64ToBytes(val *int64) []byte {

Since you are making the size and capacity a constant, you could do this instead:

func UnsafeCastInt64ToBytes(val *int64) []byte {
    return (*[8]byte)(unsafe.Pointer(val))[:]
}

No reflect.SliceHeader needed.

ianlancetaylor · 2020-10-14T19:40:56Z

Would you please share more about the "stack copier" and when an object can be moved?

Goroutines have small stacks. In the gc implementation of Go, when a goroutine needs more stack space due to additional function calls, a new stack is allocated elsewhere and the existing stack is copied into the new space. Similarly, if a quiescent goroutine has a large stack, the garbage collector will allocate a smaller memory area, copy the stack into that smaller area, and release the now-unused large stack.

Copying a stack requires adjusting all pointers to objects that live on the stack. Such pointers can only live on the stack; pointers from the heap to the stack are forbidden. (With one exception involving channel operations; that exception is handled specially.)

GingerMoon · 2020-10-15T01:10:59Z

currently we don't want to use binary.XXXEndian

Can you explain why?

The bench mark result shows that binary.xxxEndian takes 15 ns/op, while others take only 0.3 ns/op.

package main

import (
    "encoding/binary"
    "fmt"
    "reflect"
    "unsafe"
)

// Invalid usage of SliceHeader
func UnsafeCastInt64ToBytes0(val int64) []byte {
    hdr := reflect.SliceHeader{Data: uintptr(unsafe.Pointer(&val)), Len: 8, Cap: 8}
    return *(*[]byte)(unsafe.Pointer(&hdr))
}

func UnsafeCastInt64ToBytes1(val *int64) []byte {
    hdr := reflect.SliceHeader{Data: uintptr(unsafe.Pointer(val)), Len: 8, Cap: 8}
    return *(*[]byte)(unsafe.Pointer(&hdr))
}

func UnsafeCastInt64ToBytes2(val *int64) []byte {
    var result []byte
    hdr := (*reflect.SliceHeader)(unsafe.Pointer(&result))
    hdr.Data = uintptr(unsafe.Pointer(val))
    hdr.Len = 8
    hdr.Cap = 8
    return result
}

func UnsafeCastInt64ToBytes3(val *int64) []byte {
    return (*[8]byte)(unsafe.Pointer(val))[:]
}

//var result [8]byte
func UnsafeCastInt64ToBytes4(val *int64) []byte {
    var result [8]byte // if we use a global here, the benchmark result will decrease by 8 ns/op
    binary.LittleEndian.PutUint64(result[:], uint64(*val))
    return result[:]
}

//go test -bench=.
//goos: linux
//goarch: amd64
//pkg: example.com/m
//BenchmarkUnsafeCastInt64ToBytes0-8      1000000000               0.277 ns/op
//BenchmarkUnsafeCastInt64ToBytes1-8      1000000000               0.274 ns/op
//BenchmarkUnsafeCastInt64ToBytes2-8      1000000000               0.275 ns/op
//BenchmarkUnsafeCastInt64ToBytes3-8      1000000000               0.276 ns/op
//BenchmarkUnsafeCastInt64ToBytes4-8      78907146                14.8 ns/op
//PASS
//ok      example.com/m   2.414s

func main() {
    var val int64 = 0x0102
    fmt.Printf("%x \n", UnsafeCastInt64ToBytes0(val))
    fmt.Printf("%x \n", UnsafeCastInt64ToBytes1(&val))
    fmt.Printf("%x \n", UnsafeCastInt64ToBytes2(&val))
    fmt.Printf("%x \n", UnsafeCastInt64ToBytes3(&val))
    fmt.Printf("%x \n", UnsafeCastInt64ToBytes4(&val))
}

And one strange thing is, if I make UnsafeCastInt64ToBytes4.result a global, the whole bench mark is totally different:

package main

import (
    "encoding/binary"
    "fmt"
    "reflect"
    "unsafe"
)

// Invalid usage of SliceHeader
func UnsafeCastInt64ToBytes0(val int64) []byte {
    hdr := reflect.SliceHeader{Data: uintptr(unsafe.Pointer(&val)), Len: 8, Cap: 8}
    return *(*[]byte)(unsafe.Pointer(&hdr))
}

func UnsafeCastInt64ToBytes1(val *int64) []byte {
    hdr := reflect.SliceHeader{Data: uintptr(unsafe.Pointer(val)), Len: 8, Cap: 8}
    return *(*[]byte)(unsafe.Pointer(&hdr))
}

func UnsafeCastInt64ToBytes2(val *int64) []byte {
    var result []byte
    hdr := (*reflect.SliceHeader)(unsafe.Pointer(&result))
    hdr.Data = uintptr(unsafe.Pointer(val))
    hdr.Len = 8
    hdr.Cap = 8
    return result
}

func UnsafeCastInt64ToBytes3(val *int64) []byte {
    return (*[8]byte)(unsafe.Pointer(val))[:]
}

var result [8]byte
func UnsafeCastInt64ToBytes4(val *int64) []byte {
    //var result [8]byte // if we use a global here, the benchmark result will decrease by 8 ns/op
    binary.LittleEndian.PutUint64(result[:], uint64(*val))
    return result[:]
}

// If I make var result [8]byte as a global, the bench mark results are totall different!
//go test -bench=.
//goos: linux
//goarch: amd64
//pkg: example.com/m
//BenchmarkUnsafeCastInt64ToBytes0-8      1000000000               0.546 ns/op
//BenchmarkUnsafeCastInt64ToBytes1-8      1000000000               0.545 ns/op
//BenchmarkUnsafeCastInt64ToBytes2-8      1000000000               0.276 ns/op
//BenchmarkUnsafeCastInt64ToBytes3-8      1000000000               0.548 ns/op
//BenchmarkUnsafeCastInt64ToBytes4-8      1000000000               0.833 ns/op
//PASS
//ok      example.com/m   3.050s

func main() {
    var val int64 = 0x0102
    fmt.Printf("%x \n", UnsafeCastInt64ToBytes0(val))
    fmt.Printf("%x \n", UnsafeCastInt64ToBytes1(&val))
    fmt.Printf("%x \n", UnsafeCastInt64ToBytes2(&val))
    fmt.Printf("%x \n", UnsafeCastInt64ToBytes3(&val))
    fmt.Printf("%x \n", UnsafeCastInt64ToBytes4(&val))
}

And as you can see, in both cases, the implementation below is always stable as 0.27 ns/op

func UnsafeCastInt64ToBytes2(val *int64) []byte {
    var result []byte
    hdr := (*reflect.SliceHeader)(unsafe.Pointer(&result))
    hdr.Data = uintptr(unsafe.Pointer(val))
    hdr.Len = 8
    hdr.Cap = 8
    return result
}

Any insight?

randall77 · 2020-10-15T03:33:54Z

UnsafeCastInt64ToBytes4 contains an allocation, which is why it is slow. If you pass in where to write the result, then you won't need an allocation. Something like:

func UnsafeCastInt64ToBytes4(val *int64, dest []byte) {
    binary.LittleEndian.PutUint64(dest, uint64(*val))
}

or possibly

func UnsafeCastInt64ToBytes4(val *int64, dest *[8]byte) {
    binary.LittleEndian.PutUint64(dest[:], uint64(*val))
}

0.27 ns/op

That's literally 1 cycle. I think your benchmark is not measuring the correct thing. Are you using the result? The whole body of the thing you are benchmarking might be optimized away. 1 cycle is the time I get for this benchmark:

func BenchmarkFoo(b *testing.B) {
	for i := 0; i < b.N; i++ {
	}
}

BenchmarkFoo-16    	1000000000	         0.264 ns/op

GingerMoon · 2020-10-15T04:24:03Z

0.27 ns/op

That's literally 1 cycle. I think your benchmark is not measuring the correct thing. Are you using the result? The whole body of the thing you are benchmarking might be optimized away. 1 cycle is the time I get for this benchmark:

You are right that I didn't use the result in this benchmark.
But the strange thing is, as you can see above, BenchmarkUnsafeCastInt64ToBytes2 is always 0.27 ns/op

Now I did another test, I use the result in the bench mark test as below:

package main

import "testing"

func BenchmarkUnsafeCastInt64ToBytes0(b *testing.B) {
   var val int64 = 0x0102
   for i := 0; i < b.N; i++ {
       r := UnsafeCastInt64ToBytes0(val)
       r[0] = '1'
   }
}
func BenchmarkUnsafeCastInt64ToBytes1(b *testing.B) {
   var val int64 = 0x0102
   for i := 0; i < b.N; i++ {
       r := UnsafeCastInt64ToBytes1(&val)
       r[0] = '1'
   }
}
func BenchmarkUnsafeCastInt64ToBytes2(b *testing.B) {
  var val int64 = 0x0102
  for i := 0; i < b.N; i++ {
      r := UnsafeCastInt64ToBytes2(&val)
      r[0] = '1'
  }
}
func BenchmarkUnsafeCastInt64ToBytes3(b *testing.B) {
  var val int64 = 0x0102
  for i := 0; i < b.N; i++ {
      r := UnsafeCastInt64ToBytes3(&val)
      r[0] = '1'
  }
}
func BenchmarkUnsafeCastInt64ToBytes4(b *testing.B) {
   var val int64 = 0x0102
   for i := 0; i < b.N; i++ {
       r := UnsafeCastInt64ToBytes4(&val)
       r[0] = '1'
   }
}

goos: linux
goarch: amd64
pkg: example.com/m
BenchmarkUnsafeCastInt64ToBytes0-8      603495894                1.93 ns/op
BenchmarkUnsafeCastInt64ToBytes1-8      725993166                1.64 ns/op
BenchmarkUnsafeCastInt64ToBytes2-8      726476359                1.66 ns/op
BenchmarkUnsafeCastInt64ToBytes3-8      1000000000               0.278 ns/op
BenchmarkUnsafeCastInt64ToBytes4-8      73836633                14.9 ns/op
PASS
ok      example.com/m   5.536s

This time, BenchmarkUnsafeCastInt64ToBytes3-8 instead of BenchmarkUnsafeCastInt64ToBytes2-8 is always 0.27 ns/op.
Do you have any insight?

As a conclusion, below are valid cases, am I right?
Considering the performance, which way do you recommend?
(LittleEndian.PutUint64 is the slowest way)

func UnsafeCastInt64ToBytes2(val *int64) []byte {
  var result []byte
  hdr := (*reflect.SliceHeader)(unsafe.Pointer(&result))
  hdr.Data = uintptr(unsafe.Pointer(val))
  hdr.Len = 8
  hdr.Cap = 8
  return result
}

func UnsafeCastInt64ToBytes3(val *int64) []byte {
 return (*[8]byte)(unsafe.Pointer(val))[:]
}

var result [8]byte
func UnsafeCastInt64ToBytes4(val *int64) []byte {
   var result [8]byte // if we use a global here, the benchmark result will decrease by 8 ns/op
   binary.LittleEndian.PutUint64(result[:], uint64(*val))
   return result[:]
}

mdempsky · 2020-10-15T05:18:56Z

As a conclusion, below are valid cases, am I right?

Yes, UnsafeCastInt64ToBytes2 and UnsafeCastInt64ToBytes3 are both using unsafe.Pointer correctly.

Considering the performance, which way do you recommend?

I would recommend UnsafeCastInt64ToBytes3 over UnsafeCastInt64ToBytes2. UnsafeCastInt64ToBytes3 is simpler, so the compiler will have an easier time optimizing it. (That's why the compiler is still able to optimize it away in your revised benchmarks.)

I think we're getting away from the issue of whether/how to improve the package unsafe documentation though.

GingerMoon · 2020-10-15T05:30:53Z

thanks you all for the clarification! Closing the ticket.

gopherbot added the Documentation Issues describing a change to documentation. label Sep 30, 2020

bcmills added this to the Unplanned milestone Sep 30, 2020

bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 30, 2020

bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Sep 30, 2020

GingerMoon closed this as completed Oct 1, 2020

GingerMoon reopened this Oct 9, 2020

GingerMoon closed this as completed Oct 15, 2020

cuonglm mentioned this issue Nov 7, 2020

BSD mountinfo implementation is unsound moby/sys#51

Open

golang locked and limited conversation to collaborators Oct 15, 2021

gopherbot added the FrozenDueToAge label Oct 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

reflect: documentation about "Conversion of a reflect.SliceHeader or reflect.StringHeader Data field to or from Pointer" is misleading #41705

reflect: documentation about "Conversion of a reflect.SliceHeader or reflect.StringHeader Data field to or from Pointer" is misleading #41705

GingerMoon commented Sep 30, 2020 •

edited

Loading

bcmills commented Sep 30, 2020 •

edited

Loading

bcmills commented Sep 30, 2020

bcmills commented Sep 30, 2020

mdempsky commented Sep 30, 2020

GingerMoon commented Sep 30, 2020 •

edited

Loading

mdempsky commented Sep 30, 2020

GingerMoon commented Sep 30, 2020 •

edited

Loading

mdempsky commented Sep 30, 2020 •

edited

Loading

GingerMoon commented Oct 1, 2020

ianlancetaylor commented Oct 1, 2020

GingerMoon commented Oct 1, 2020

GingerMoon commented Oct 9, 2020

ianlancetaylor commented Oct 9, 2020

bcmills commented Oct 9, 2020

GingerMoon commented Oct 13, 2020

ianlancetaylor commented Oct 13, 2020

GingerMoon commented Oct 13, 2020

ianlancetaylor commented Oct 13, 2020

GingerMoon commented Oct 14, 2020

bcmills commented Oct 14, 2020

randall77 commented Oct 14, 2020

ianlancetaylor commented Oct 14, 2020

GingerMoon commented Oct 15, 2020

randall77 commented Oct 15, 2020

GingerMoon commented Oct 15, 2020

mdempsky commented Oct 15, 2020

GingerMoon commented Oct 15, 2020

reflect: documentation about "Conversion of a reflect.SliceHeader or reflect.StringHeader Data field to or from Pointer" is misleading #41705

reflect: documentation about "Conversion of a reflect.SliceHeader or reflect.StringHeader Data field to or from Pointer" is misleading #41705

Comments

GingerMoon commented Sep 30, 2020 • edited Loading

bcmills commented Sep 30, 2020 • edited Loading

bcmills commented Sep 30, 2020

bcmills commented Sep 30, 2020

mdempsky commented Sep 30, 2020

GingerMoon commented Sep 30, 2020 • edited Loading

mdempsky commented Sep 30, 2020

GingerMoon commented Sep 30, 2020 • edited Loading

mdempsky commented Sep 30, 2020 • edited Loading

GingerMoon commented Oct 1, 2020

ianlancetaylor commented Oct 1, 2020

GingerMoon commented Oct 1, 2020

GingerMoon commented Oct 9, 2020

ianlancetaylor commented Oct 9, 2020

bcmills commented Oct 9, 2020

GingerMoon commented Oct 13, 2020

ianlancetaylor commented Oct 13, 2020

GingerMoon commented Oct 13, 2020

ianlancetaylor commented Oct 13, 2020

GingerMoon commented Oct 14, 2020

bcmills commented Oct 14, 2020

randall77 commented Oct 14, 2020

ianlancetaylor commented Oct 14, 2020

GingerMoon commented Oct 15, 2020

randall77 commented Oct 15, 2020

GingerMoon commented Oct 15, 2020

mdempsky commented Oct 15, 2020

GingerMoon commented Oct 15, 2020

GingerMoon commented Sep 30, 2020 •

edited

Loading

bcmills commented Sep 30, 2020 •

edited

Loading

GingerMoon commented Sep 30, 2020 •

edited

Loading

GingerMoon commented Sep 30, 2020 •

edited

Loading

mdempsky commented Sep 30, 2020 •

edited

Loading