cmd/buildlet, cmd/makemac: normalize macstadium host names for monitoring

In prep for better alerting when dedicated (reverse) buildlers
disappear, normalize the the MacStadium host names to remove the
extraneous guest OS version from them, so we can track the host's
last-seen time reliably over time, even as the guest OS version
changes.

This CL also cleans up makemac while it's there and fixes some bugs
and adds some paranoia checks and cleans up logging and adds an HTTP
status handler.

A future change will improve coordinator monitoring of reverse
buildlets.

Updates golang/go#21315

Change-Id: I3d09168cc91f37715b65ae2924a1642401e18808
Reviewed-on: https://go-review.googlesource.com/53353
Reviewed-by: Jessie Frazelle <[email protected]>

Author: bradfitz

cmd/buildlet/buildlet.go

@@ -1497,7 +1497,31 @@ func configureMacStadium() {
 	}
 	*reverse = "darwin-amd64-" + major + "_" + minor
 	*coordinator = "farmer.golang.org:443"
-	*hostname = vmwareGetInfo("guestinfo.name")
+
+	// guestName is set by cmd/makemac to something like
+	// "mac_10_10_host01b" or "mac_10_12_host01a", which encodes
+	// three things: the mac OS version of the guest VM, which
+	// physical machine it's on (1 to 10, currently) and which of
+	// two possible VMs on that host is running (a or b). For
+	// monitoring purposes, we want stable hostnames and don't
+	// care which OS version is currently running (which changes
+	// constantly), so normalize these to only have the host
+	// number and side (a or b), without the OS version. The
+	// buildlet will report the OS version to the coordinator
+	// anyway. We could in theory do this normalization in the
+	// coordinator, but we don't want to put buildlet-specific
+	// knowledge there, and this file already contains a bunch of
+	// buildlet host-specific configuration, so normalize it here.
+	guestName := vmwareGetInfo("guestinfo.name") // "mac_10_12_host01a"
+	hostPos := strings.Index(guestName, "_host")
+	if hostPos == -1 {
+		// Assume cmd/makemac changed its conventions.
+		// Maybe all this normalization belongs there anyway,
+		// but normalizing here is a safer first step.
+		*hostname = guestName
+	} else {
+		*hostname = "macstadium" + guestName[hostPos:] // "macstadium_host01a"
+	}
 }
 
 func disableMacScreensaver() {

cmd/makemac/README.md

@@ -1,7 +1,19 @@
-<!-- Auto-generated by x/build/update-readmes.go -->
-
 [![GoDoc](https://godoc.org/golang.org/x/build/cmd/makemac?status.svg)](https://godoc.org/golang.org/x/build/cmd/makemac)
 
 # golang.org/x/build/cmd/makemac
 
 The makemac command starts OS X VMs for the builders.
+
+## Deploying
+
+```
+* On Linux,
+  $ go install golang.org/x/build/cmd/makemac
+  $ scp -i ~/.ssh/id_ed25519_golang1 $GOPATH/bin/makemac [email protected]:makemac.new
+  $ ssh -i ~/.ssh/id_ed25519_golang1 [email protected] 'cp makemac makemac.old; install makemac.new makemac'
+  $ ssh -i ~/.ssh/id_ed25519_golang1 [email protected]
+
+On that host,
+  * sudo systemctl restart makemac
+  $ sudo journalctl -f -u makemac     # watch it
+```