all: test that all examples run on golang.org, i.e. the playground

[mvdan]

Right now, examples are only run via go test by the trybots. That checks whether or not they run as tests, but it doesn't check if they run in the playground.

For instance, see #19823. That example depends on its own test file, example_test.go. That will be available if one runs go test, but not if one presses "Run" in https://golang.org/pkg/go/parser/#example_ParseFile which will run it in the playground.

There should be a mechanism to ensure that all of the examples run on the golang.org docs too. In other words, that they pass when executed in the playground.

[mvdan]

Perhaps there could be a separate trybot for this, similar to how there are race trybots. Not sure how the playground could be hooked up - not familiar enough with it.

CC @josharian

[josharian]

We have a nacl builder, which should run the examples. That should cover this, although I'm obviously wrong about that somehow.

[mvdan]

But I understand the nacl builder just does go test like the rest, correct?

[josharian]

go test runs examples, though.

[mvdan]

Yes, but if I understand correctly the nacl builder will still have example_test.go in the current directory when it runs that particular example via go test.

[josharian]

I'm still obviously wrong somewhere, but each executed nacl process should not have access to the filesystem once started.

[mvdan]

Ah, that makes sense and should do the trick. But it obviously doesn't :)

Perhaps we should add a test that only runs on nacl that checks that its own test file doesn't exist in disk, or that it cannot be opened?

[josharian]

I'd say the first step is figuring out exactly why #19823 was passing on nacl.

[tzneal]

As to why it passes, src/naclmake.bash packages up a fake file system for nacl tests according to misc/nacl/testzip.proto. On line 127 there it includes everything beneath src/go/parser which includes example_test.go.

Maybe the playground should use the same packaged filesystem?

[ysmolski]

Do we know how many examples fail to run on playground?
@jimmyfrasche

[jimmyfrasche]

This isn't up to date, but I did a run through earlier this year: #9679 (comment)

Since there have been improvements to how go/doc extracts examples, allowing more to be playable, some fixed examples, and more examples overall, someone's just going to need to run through each example and hit play (preferably a trybot).

I don't have a count but it's more than 1, so it's enough to warrant attention and action.