cmd/compile/internal/types2: invalid recursive type not detected and panics

[qmuntal]

What version of Go are you using (go version)?

go version devel go1.18-67f7e16 Fri Aug 27 08:01:41 2021 +0000 windows/amd64

Does this issue reproduce with the latest release?

tip only

What operating system and processor architecture are you using (go env)?

go env Output

set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\muntal\AppData\Local\go-build
set GOENV=C:\Users\muntal\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\muntal\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\muntal\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Users\muntal\sdk\gotip
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Users\muntal\sdk\gotip\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=devel go1.18-67f7e16 Fri Aug 27 08:01:41 2021 +0000
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\muntal\AppData\Local\Temp\go-build2947247836=/tmp/go-build -gno-record-gcc-switches

What did you do?

type P0[T1, T2 any] struct {
  F *P0[P0[T1, T2], T2] // INVALID; must be [T1, T2]
}

go build .

What did you expect to see?

A compiler error saying that this recursion is not valid, as per the type parameters proposal:

A generic type can refer to itself in cases where a type can ordinarily refer to itself, but when it does so the type arguments must be the type parameters, listed in the same order.

What did you see instead?

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x5c pc=0x57dbe5]

goroutine 1 [running]:
cmd/compile/internal/types.(*Type).SetUnderlying(0xc0003d0850, 0x0)
        C:/_/go/src/cmd/compile/internal/types/type.go:1745 +0x25
cmd/compile/internal/noder.(*irgen).typ0(0xc000146200, {0xe3ddf0, 0xc000164510})
        C:/_/go/src/cmd/compile/internal/noder/types.go:166 +0x1499
cmd/compile/internal/noder.(*irgen).typ1(0xc000146200, {0xe3ddf0, 0xc000164510})
        C:/_/go/src/cmd/compile/internal/noder/types.go:61 +0x68
cmd/compile/internal/noder.(*irgen).typ0(0xc000146200, {0xe3de18, 0xc00005cef0})
        C:/_/go/src/cmd/compile/internal/noder/types.go:192 +0x5bd
cmd/compile/internal/noder.(*irgen).typ1(0xc000146200, {0xe3de18, 0xc00005cef0})
        C:/_/go/src/cmd/compile/internal/noder/types.go:61 +0x68
cmd/compile/internal/noder.(*irgen).typ0(0xc000146200, {0xe3de90, 0xc0003bfd70})
        C:/_/go/src/cmd/compile/internal/noder/types.go:202 +0x169b
cmd/compile/internal/noder.(*irgen).typ1(0xc000146200, {0xe3de90, 0xc0003bfd70})
        C:/_/go/src/cmd/compile/internal/noder/types.go:61 +0x68
cmd/compile/internal/noder.(*irgen).typ0(0xc000146200, {0xe3ddf0, 0xc000164480})
        C:/_/go/src/cmd/compile/internal/noder/types.go:166 +0x1489
cmd/compile/internal/noder.(*irgen).typ1(0xc000146200, {0xe3ddf0, 0xc000164480})
        C:/_/go/src/cmd/compile/internal/noder/types.go:61 +0x68
cmd/compile/internal/noder.(*irgen).typ0(0xc000146200, {0xe3de18, 0xc00005cf10})
        C:/_/go/src/cmd/compile/internal/noder/types.go:192 +0x5bd
cmd/compile/internal/noder.(*irgen).typ1(0xc000146200, {0xe3de18, 0xc00005cf10})
        C:/_/go/src/cmd/compile/internal/noder/types.go:61 +0x68
cmd/compile/internal/noder.(*irgen).typ0(0xc000146200, {0xe3de90, 0xc0003bfda0})
        C:/_/go/src/cmd/compile/internal/noder/types.go:202 +0x169b
cmd/compile/internal/noder.(*irgen).typ1(0xc000146200, {0xe3de90, 0xc0003bfda0})
        C:/_/go/src/cmd/compile/internal/noder/types.go:61 +0x68
cmd/compile/internal/noder.(*irgen).typ0(0xc000146200, {0xe3ddf0, 0xc0001643f0})
        C:/_/go/src/cmd/compile/internal/noder/types.go:166 +0x1489
cmd/compile/internal/noder.(*irgen).typ1(0xc000146200, {0xe3ddf0, 0xc0001643f0})
        C:/_/go/src/cmd/compile/internal/noder/types.go:61 +0x68
cmd/compile/internal/noder.(*irgen).typ0(0xc000146200, {0xe3de18, 0xc00005cc80})
        C:/_/go/src/cmd/compile/internal/noder/types.go:192 +0x5bd
cmd/compile/internal/noder.(*irgen).typ1(0xc000146200, {0xe3de18, 0xc00005cc80})
        C:/_/go/src/cmd/compile/internal/noder/types.go:61 +0x68
cmd/compile/internal/noder.(*irgen).typ0(0xc000146200, {0xe3de90, 0xc0003bf5f0})
        C:/_/go/src/cmd/compile/internal/noder/types.go:202 +0x169b
cmd/compile/internal/noder.(*irgen).typ1(0xc000146200, {0xe3de90, 0xc0003bf5f0})
        C:/_/go/src/cmd/compile/internal/noder/types.go:61 +0x68
cmd/compile/internal/noder.(*irgen).typ(0xc000146200, {0xe3de90, 0xc0003bf5f0})
        C:/_/go/src/cmd/compile/internal/noder/types.go:36 +0x2f
cmd/compile/internal/noder.(*irgen).expr(0xc000146200, {0xe3f048, 0xc000033680})
        C:/_/go/src/cmd/compile/internal/noder/expr.go:46 +0x95e
cmd/compile/internal/noder.(*irgen).typeExpr(0xc000146200, {0xe3f048, 0xc000033680})
        C:/_/go/src/cmd/compile/internal/noder/expr.go:487 +0x31
cmd/compile/internal/noder.(*irgen).typeDecl(0xc000146200, 0xc0001241b0, 0xc0000698c0)
        C:/_/go/src/cmd/compile/internal/noder/decl.go:187 +0x105
cmd/compile/internal/noder.(*irgen).generate(0xc000146200, {0xc000006450, 0x1, 0x15864c6df87})
        C:/_/go/src/cmd/compile/internal/noder/irgen.go:248 +0x6c8
cmd/compile/internal/noder.check2({0xc000006450, 0x1, 0x1})
        C:/_/go/src/cmd/compile/internal/noder/irgen.go:92 +0x177
cmd/compile/internal/noder.LoadPackage({0xc00005e3d0, 0x1, 0x0})
        C:/_/go/src/cmd/compile/internal/noder/noder.go:90 +0x335
cmd/compile/internal/gc.Main(0xd07798)
        C:/_/go/src/cmd/compile/internal/gc/main.go:190 +0xaf3
main.main()
        C:/_/go/src/cmd/compile/main.go:55 +0xdd

@griesemer

[qmuntal]

The following non-compliant type definitions do not cause a panic but are accepted by the compiler:

type P[T1, T2 any] struct {
  F *P[T2, T1] // INVALID; must be [T1, T2]
}

type P[T1, T2 any] struct {
  F *P[int, int] // INVALID; must be [T1, T2]
}

[griesemer]

There are two issues here:

1. The type checker currently does not detect those cases but silently accepts them (and doesn't crash).
   See types2, go/types: need to detect invalid recursive type instantiation #48098.
2. There may be a bug in the compiler's front-end (after type-checking).

cc: @danscales

[griesemer]

Closing as duplicate of #48098.