x/pkgsite: don't show un-runnable examples as runnable

[adg]

The example for exec.Command won't run in the playground sandbox because it depends on 'tr'.

http://golang.org/pkg/os/exec/#Command

There are other such examples. We should probably detect (somehow) that they won't run, and not display them as runnable examples on golang.org.

[jimmyfrasche]

Did a quick run through of stdlib examples.

There were some I couldn't check because #23095 seems to effect quite a few examples.

Other than os/exec, the patterns I saw were:

A lot of tests aren't playable due to files not existing. For the go/* packages this would be fixed by #23603. For the rest that use files, they appear to fall into two categories:

1. Explicitly assuming the file doesn't exist and it just won't run since it's the playground
2. Implicitly assuming the example will be executed with the current working directory set to the package directory (a lot of os.Open("testdata/afile"))

https://golang.org/pkg/crypto/x509/#example_Certificate_Verify runs locally but fails on the playground with "certificate has expired or is not yet valid", presumably because of the fake time.

https://golang.org/pkg/log/syslog/#example_Dial doesn't run because syslog.Dial isn't defined on nacl

I'm 99.999% sure this example deadlocking is a bug but will leave it up to the experts: https://golang.org/pkg/net/#example_Listener Similarly for https://golang.org/pkg/net/http/#example_Get replying "Get http://www.google.com/robots.txt: dial tcp: Protocol not available"

Some of these examples have output and some don't. If they all did that could be used to build a static list of non-runnable examples. (But then it's not clear if they're not runnable due to programmer error or playground limitations).

Given the variety and subtlety among the examples that don't run, maybe an annotation like //godoc: noplay would be the way to go?

If the examples that cannot ever be run on the playground are tagged as such, there could probably be some sort of playground trybot that makes sure examples match their output when run on the playground.

[vbmade2000]

@broady @bradfitz Is anyone working on this issue ? I can work otherwise.

[bradfitz]

Not that I know of.

[vbmade2000]

Cool. I'll create CL.

[gopherbot]

Change https://golang.org/cl/122237 mentions this issue: path/filepath: make Walk example runnable in the playground

[mostynb]

Also, is it worth minimizing this issue by making more examples runnable? For example: https://golang.org/cl/122237

[gopherbot]

Change https://golang.org/cl/145838 mentions this issue: crypto/cipher: make stream examples runnable in the playground

[gopherbot]

Change https://golang.org/cl/146118 mentions this issue: go/doc: inspect result types when synthesising playground examples

[ysmolski]

64 examples out of 610 in go/src/... won't run in Playground.
Also another 6 examples are marked as non-playable.
Just a few of them can be fixed so they will run in Playground, but definitely not all of them.

How can we mark the examples that should not be played?

EDIT: I mean that those 64 examples [when ran] return errors unrelated to their code. Basically they are broken for the user. I am sure they confuse people here and there.

[jimmyfrasche]

Ideally, every example should be runnable on the playground. Any that can possibly be rewritten to be runnable should be, of course.

Since this is playground specific, it shouldn't be in go/doc.

That leaves a comment that godoc recognizes. Something like //playground: no.

[ysmolski]

Posting the results of running all the examples against the playground. Most of them are net or os failures. I wonder how hard it would be to create fake network for the playground.

Some examples take too long, like time.Tick, but that should be okay when new version godoc will be deployed. It will show output even when the example has timed out.

Feel free to make any of those examples playground-friendly.

$GOROOT/src/archive/zip/example_test.go:50:22 ExampleReader
	stderr: 2009/11/10 23:00:00 open testdata/readme.zip: No such file or directory

$GOROOT/src/crypto/md5/example_test.go:29:24 ExampleNew_file
	stderr: 2009/11/10 23:00:00 open file.txt: No such file or directory

$GOROOT/src/crypto/sha1/example_test.go:29:24 ExampleNew_file
	stderr: 2009/11/10 23:00:00 open file.txt: No such file or directory

$GOROOT/src/crypto/sha256/example_test.go:28:24 ExampleNew_file
	stderr: 2009/11/10 23:00:00 open file.txt: No such file or directory

$GOROOT/src/crypto/tls/example_test.go:28:20 ExampleDial
	stderr: panic: failed to connect: dial tcp: Protocol not available

$GOROOT/src/crypto/tls/example_test.go:118:31 ExampleLoadX509KeyPair
	stderr: 2009/11/10 23:00:00 open testdata/example-cert.pem: No such file or directory

$GOROOT/src/crypto/tls/example_test.go:160:38 ExampleX509KeyPair_httpServer
	stderr: fatal error: all goroutines are asleep - deadlock!

$GOROOT/src/crypto/x509/example_test.go:16:34 ExampleCertificate_Verify
	stderr: panic: failed to verify certificate: x509: certificate has expired or is not yet valid

$GOROOT/src/database/sql/example_test.go:140:28 ExampleConn_BeginTx
	stderr: panic: runtime error: invalid memory address or nil pointer dereference

$GOROOT/src/database/sql/example_test.go:156:32 ExampleConn_ExecContext
	stderr: panic: runtime error: invalid memory address or nil pointer dereference

$GOROOT/src/database/sql/example_test.go:55:30 ExampleDB_ExecContext
	stderr: panic: runtime error: invalid memory address or nil pointer dereference

$GOROOT/src/database/sql/example_test.go:132:30 ExampleDB_PingContext
	stderr: panic: runtime error: invalid memory address or nil pointer dereference

$GOROOT/src/database/sql/example_test.go:19:31 ExampleDB_QueryContext
	stderr: panic: runtime error: invalid memory address or nil pointer dereference

$GOROOT/src/database/sql/example_test.go:40:34 ExampleDB_QueryRowContext
	stderr: panic: runtime error: invalid memory address or nil pointer dereference

$GOROOT/src/database/sql/example_test.go:70:43 ExampleDB_Query_multipleResultSets
	stderr: panic: runtime error: invalid memory address or nil pointer dereference

$GOROOT/src/database/sql/example_test.go:262:20 ExampleRows
	stderr: panic: runtime error: invalid memory address or nil pointer dereference

$GOROOT/src/database/sql/example_test.go:221:20 ExampleStmt
	stderr: panic: runtime error: invalid memory address or nil pointer dereference

$GOROOT/src/database/sql/example_test.go:242:36 ExampleStmt_QueryRowContext
	stderr: panic: runtime error: invalid memory address or nil pointer dereference

$GOROOT/src/database/sql/example_test.go:178:30 ExampleTx_ExecContext
	stderr: panic: runtime error: invalid memory address or nil pointer dereference

$GOROOT/src/database/sql/example_test.go:196:27 ExampleTx_Rollback
	stderr: panic: runtime error: invalid memory address or nil pointer dereference

$GOROOT/src/go/types/example_test.go:115:25 ExampleMethodSet
	stderr: 2009/11/10 23:00:00 celsius.go:3:8: could not import fmt (can't find import: "fmt")

$GOROOT/src/go/types/example_test.go:35:21 ExampleScope
	stderr: 2009/11/10 23:00:00 main.go:3:8: could not import fmt (can't find import: "fmt")

$GOROOT/src/image/png/example_test.go:24:22 ExampleDecode
	Errors: prog.go:15:18: undefined: io

$GOROOT/src/io/ioutil/example_test.go:91:24 ExampleReadFile
	stderr: 2009/11/10 23:00:00 open testdata/hello: No such file or directory

$GOROOT/src/io/ioutil/example_test.go:103:25 ExampleWriteFile
	stderr: 2009/11/10 23:00:00 open testdata/hello: No such file or directory

$GOROOT/src/log/syslog/example_test.go:15:20 ExampleDial
	Errors: prog.go:10:17: undefined: syslog.Dial

$GOROOT/src/net/example_test.go:14:24 ExampleListener
	stderr: fatal error: all goroutines are asleep - deadlock!

$GOROOT/src/net/example_test.go:123:31 ExampleUDPConn_WriteTo
	stderr: 2009/11/10 23:00:00 write udp 0.0.0.0:2->192.0.2.1:2000: sendto: Connection refused

$GOROOT/src/net/http/example_filesystem_test.go:5:1 ExampleFileServer_dotFileHiding
	stderr: fatal error: all goroutines are asleep - deadlock!

$GOROOT/src/net/http/example_handle_test.go:5:1 ExampleHandle
	stderr: fatal error: all goroutines are asleep - deadlock!

$GOROOT/src/net/http/example_test.go:57:26 ExampleFileServer
	stderr: fatal error: all goroutines are asleep - deadlock!

$GOROOT/src/net/http/example_test.go:44:19 ExampleGet
	stderr: 2009/11/10 23:00:00 Get http://www.google.com/robots.txt: dial tcp: Protocol not available

$GOROOT/src/net/http/example_test.go:163:26 ExampleHandleFunc
	stderr: fatal error: all goroutines are asleep - deadlock!

$GOROOT/src/net/http/example_test.go:152:30 ExampleListenAndServe
	stderr: fatal error: all goroutines are asleep - deadlock!

$GOROOT/src/net/http/example_test.go:141:33 ExampleListenAndServeTLS
	stderr: 2009/11/10 23:00:00 About to listen on 8443. Go to https://127.0.0.1:8443/

$GOROOT/src/net/http/example_test.go:183:31 ExampleNotFoundHandler
	stderr: fatal error: all goroutines are asleep - deadlock!

$GOROOT/src/net/http/example_test.go:116:31 ExampleServer_Shutdown
	Errors: process took too long

$GOROOT/src/net/http/httptrace/example_test.go:14:16 Example
	stderr: 2009/11/10 23:00:00 dial tcp: Protocol not available

$GOROOT/src/net/smtp/example_test.go:13:16 Example
	stderr: 2009/11/10 23:00:00 dial tcp: Protocol not available

$GOROOT/src/net/smtp/example_test.go:57:25 ExamplePlainAuth
	stderr: 2009/11/10 23:00:00 dial tcp: Protocol not available

$GOROOT/src/net/smtp/example_test.go:68:24 ExampleSendMail
	stderr: 2009/11/10 23:00:00 dial tcp: Protocol not available

$GOROOT/src/os/example_test.go:38:21 ExampleChmod
	stderr: 2009/11/10 23:00:00 chmod some-filename: No such file or directory

$GOROOT/src/os/example_test.go:44:23 ExampleChtimes
	stderr: 2009/11/10 23:00:00 chtimes some-filename: No such file or directory

$GOROOT/src/os/example_test.go:52:24 ExampleFileMode
	stderr: 2009/11/10 23:00:00 lstat some-filename: No such file or directory

$GOROOT/src/os/exec/example_test.go:139:34 ExampleCmd_CombinedOutput
	stderr: 2009/11/10 23:00:00 exec: "sh": executable file not found in $PATH

$GOROOT/src/os/exec/example_test.go:52:26 ExampleCmd_Output
	stderr: 2009/11/10 23:00:00 exec: "date": executable file not found in $PATH

$GOROOT/src/os/exec/example_test.go:60:23 ExampleCmd_Run
	stderr: 2009/11/10 23:00:00 Running command and waiting for it to finish...

$GOROOT/src/os/exec/example_test.go:67:25 ExampleCmd_Start
	stderr: 2009/11/10 23:00:00 exec: "sleep": executable file not found in $PATH

$GOROOT/src/os/exec/example_test.go:120:30 ExampleCmd_StderrPipe
	stderr: 2009/11/10 23:00:00 exec: "sh": executable file not found in $PATH

$GOROOT/src/os/exec/example_test.go:100:29 ExampleCmd_StdinPipe
	stderr: 2009/11/10 23:00:00 exec: "cat": executable file not found in $PATH

$GOROOT/src/os/exec/example_test.go:78:30 ExampleCmd_StdoutPipe
	stderr: 2009/11/10 23:00:00 exec: "echo": executable file not found in $PATH

$GOROOT/src/os/exec/example_test.go:29:23 ExampleCommand
	stderr: 2009/11/10 23:00:00 exec: "tr": executable file not found in $PATH

$GOROOT/src/os/exec/example_test.go:41:35 ExampleCommand_environment
	stderr: 2009/11/10 23:00:00 exec: "prog": executable file not found in $PATH

$GOROOT/src/os/exec/example_test.go:21:24 ExampleLookPath
	stderr: 2009/11/10 23:00:00 installing fortune is in your future

$GOROOT/src/os/signal/example_test.go:13:22 ExampleNotify
	Errors: process took too long

$GOROOT/src/os/signal/example_test.go:25:33 ExampleNotify_allSignals
	Errors: process took too long

$GOROOT/src/runtime/trace/example_test.go:5:1 Example
	Errors: process took too long

$GOROOT/src/syscall/js/js_test.go:343:27 ExampleNewCallback
	Errors: prog.go:5:2: build constraints exclude all Go files in /usr/local/go/src/syscall/js

$GOROOT/src/syscall/syscall_windows_test.go:57:27 ExampleLoadLibrary
	Errors: prog.go:12:12: undefined: syscall.LoadLibrary

$GOROOT/src/testing/benchmark_test.go:97:29 ExampleB_RunParallel
	Errors: process took too long

$GOROOT/src/time/example_test.go:146:20 ExampleTick
	Errors: process took too long