Update thing-flinger to track changes to omicron (#1001)

Summary of changes:

* Improve error messages in omicron-package
* Teach `thing-flinger sync` about `config-rss.toml`
* Add `-y` option to `install_prerequisites.sh` to skip confirm prompts
* Add `-p` option to `install_prerequisites.sh` to skip PATH check
* Add `thing-flinger install-prereqs` subcommand

Author: jgallagher

deploy/README.adoc

@@ -83,42 +83,58 @@ all the dependencies for Omicron installed. Following the *prerequisites* in the
 https://github.com/oxidecomputer/omicron/#build-and-run[Build and run] section of the main Omicron
 README is probably a good idea.
 
-=== Command Based Workflow
+==== Update `config-rss.toml`
 
-==== Build thing-flinger on client
-`thing-flinger` is part of the `omicron-package` crate.
+Currently rack setup is driven by a configuration file that lives at
+`smf/sled-agent/config-rss.toml` in the root of this repository. The committed
+configuration of that file contains a single `[[requests]]` entry (with many
+services inside it), which means it will start services on only one sled. To
+start services (e.g., nexus) on multiple sleds, add additional entries to that
+configuration file before proceeding.
 
-`cargo build -p omicron-package`
+=== Command Based Workflow
 
 ==== sync
-Copy your source code to the builder. Note that this copies over your `.git` subdirectory on purpose so
-that a branch can be configured for building with the `git_treeish` field in the toml `builder`
-table.
+Copy your source code to the builder.
+
+`cargo run --bin thing-flinger -- -c <CONFIG> sync`
 
-`./target/debug/thing-flinger -c <CONFIG.toml> sync`
+==== Install Prerequisites
+Install necessary build and runtime dependencies (including downloading prebuilt
+binaries like Clickhouse and CockroachDB) on the builder and all deployment
+targets. This step only needs to be performed once, absent any changes to the
+dependencies, but is idempotent so may be run multiple times.
 
-==== build-minimal
-Build necessary parts of omicron on the builder, required for future use by thing-flinger.
+`cargo run --bin thing-flinger -- -c <CONFIG> install-prereqs`
 
-`./target/debug/thing-flinger -c <CONFIG> build-minimal`
+==== check (optional)
+Run `cargo check` on the builder against the copy of `omicron` that was sync'd
+to it in the previous step.
 
-==== package 
+`cargo run --bin thing-flinger -- -c <CONFIG> build check`
+
+==== package
 Build and package omicron using `omicron-package` on the builder.
 
-`./target/debug/thing-flinger -c <CONFIG> package`
+`cargo run --bin thing-flinger -- -c <CONFIG> build package`
 
 ==== overlay
 Create files that are unique to each deployment server.
 
-`./target/debug/thing-flinger -c <CONFIG> overlay`
+`cargo run --bin thing-flinger -- -c <CONFIG> overlay`
 
-==== install 
+==== install
 Install omicron to all machines, in parallel. This consists of copying the packaged omicron tarballs
 along with overlay files, and omicron-package and its manifest to a `staging` directory on each
 deployment server, and then running omicron-package, installing overlay files, and restarting
 services.
 
-`./target/debug/thing-flinger -c <CONFIG> install`
+`cargo run --bin thing-flinger -- -c <CONFIG> deploy install`
+
+==== uninstall
+Uninstall omicron from all machines.
+
+`cargo run --bin thing-flinger -- -c <CONFIG> deploy uninstall`
 
 === Current Limitations
 
@@ -140,3 +156,67 @@ effort to use securely. This particular implementation wraps the openssh ssh cli
 `std::process::Command`, rather than using the `ssh2` crate, because ssh2, as a wrapper around
 `libssh`, does not support agent-forwarding.
 
+== Notes on Using VMs as Deployed Servers on a Linux Host
+
+TODO: This section should be fleshed out more and potentially lifted to its own
+document; for now this is a collection of rough notes.
+
+---
+
+It's possible to use a Linux libvirt host running multiple helios VMs as the
+builder/deployment server targets, but it requires some additional setup beyond
+[`helios-engvm`](https://github.com/oxidecomputer/helios-engvm).
+
+`thing-flinger` does not have any support for running the
+`tools/create_virtual_hardware.sh` script; this will need to be done by hand on
+each VM.
+
+---
+
+To enable communication between the VMs over their IPv6 bootstrap networks:
+
+1. Enable IPv6 and DHCP on the virtual network libvirt uses for the VMs; e.g.,
+
+```xml
+  <ip family="ipv6" address="fdb0:5254::1" prefix="96">
+    <dhcp>
+      <range start="fdb0:5254::100" end="fdb0:5254::1ff"/>
+    </dhcp>
+  </ip>
+```
+
+After booting the VMs with this enabled, they should be able to ping each other
+over their acquired IPv6 addresses, but connecting to each other over the
+`bootstrap6` interface that sled-agent creates will fail.
+
+2. Explicitly add routes in the Linux host for the `bootstrap6` addresses,
+specifying the virtual interface libvirt created that is used by the VMs.
+
+```
+bash% sudo ip -6 route add fdb0:5254:13:7331::1/64 dev virbr1
+bash% sudo ip -6 route add fdb0:5254:f0:acfd::1/64 dev virbr1
+```
+
+3. Once the sled-agents advance sufficiently to set up `sled6` interfaces,
+routes need to be added for them both in the Linux host and in the Helios VMs.
+Assuming two sleds with these interfaces:
+
+```
+# VM 1
+vioif0/sled6      static   ok           fd00:1122:3344:1::1/64
+# VM 2
+vioif0/sled6      static   ok           fd00:1122:3344:2::1/64
+```
+
+The Linux host needs to be told to route that subnet to the appropriate virtual
+interface:
+
+```
+bash% ip -6 route add fd00:1122:3344::1/48 dev virbr1
+```
+
+and each Helios VM needs to be told to route that subnet to the host gateway:
+
+```
+vm% pfexec route add -inet6 fd00:1122:3344::/48 $IPV6_HOST_GATEWAY_ADDR
+```

deploy/src/bin/deployment-example.toml

@@ -15,7 +15,9 @@ server = "foo"
 omicron_path = "/remote/path/to/omicron"
 
 [deployment]
-servers = ["foo", "bar"]
+# which server is responsible for running the rack setup service; must
+# refer to one of the `servers` in the servers table
+rss_server = "foo"
 rack_secret_threshold = 2
 # Location where files to install will be placed before running
 # `omicron-package install`