Repository navigation
Add new multirack CRDB tables for unadopted racks and sleds #11548
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
39498f0
eb2ebaf
bdbe9dd
0b0ba42
b1bbed4
d3844c6
eede59c
42cf02d
324d548
8b69ba5
3989c3e
2a6ffe6
cefd279
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| -- Tracks other racks that have joined the underlay network but haven't been | ||
| -- adopted into the multirack cluster. | ||
| -- | ||
| -- If/when a rack is adopted, its row in this table should be marked deleted, | ||
| -- and a corresponding row should be inserted into the `rack` table. | ||
| CREATE TABLE IF NOT EXISTS omicron.public.unadopted_rack ( | ||
| -- This should be the same id that gets used for the `rack` table, as well | ||
| -- as the `rack_id` column in `unadopted_sled`: | ||
| id UUID PRIMARY KEY, | ||
|
|
||
| time_created TIMESTAMPTZ NOT NULL, | ||
| time_modified TIMESTAMPTZ NOT NULL, | ||
| time_deleted TIMESTAMPTZ, | ||
|
|
||
| -- The IPv6 underlay /56 prefix for the rack: | ||
| rack_subnet INET NOT NULL, | ||
|
|
||
| -- Socket address of the multirack join service running on a sled-agent in | ||
| -- the unadopted rack: | ||
| join_service_ip INET NOT NULL, | ||
| join_service_port INT4 | ||
| CHECK (join_service_port BETWEEN 0 AND 65535) | ||
| NOT NULL | ||
| ) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| -- Tracks sleds from other racks that have joined the underlay network but | ||
| -- haven't been adopted into the multirack cluster. | ||
| -- | ||
| -- If/when these sleds are adopted, these rows should be marked deleted, and | ||
| -- there will subsequently be corresponding rows created in the `sled` and | ||
| -- `inv_sled_agent` tables via the normal mechanisms. | ||
| CREATE TABLE IF NOT EXISTS omicron.public.unadopted_sled ( | ||
| id UUID PRIMARY KEY, | ||
| rack_id UUID NOT NULL, | ||
|
andrewjstone marked this conversation as resolved.
Resolved
|
||
|
|
||
| time_created TIMESTAMPTZ NOT NULL, | ||
| time_modified TIMESTAMPTZ NOT NULL, | ||
| time_deleted TIMESTAMPTZ, | ||
|
|
||
| -- Baseboard information about the sled | ||
| serial_number STRING(63) NOT NULL, | ||
| part_number STRING(63) NOT NULL, | ||
|
|
||
| ip INET NOT NULL | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd call this
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We may want to provide a status about whether the sled-agent is running. It depends on how we're planning to use this. If we are planning to decouple the operator request for listing unadopted racks and their sleds from the request to get the status of the join service by just reading this table, we should have the latest polled status. I added support for the start status in #11535. I think that's probably the right direction. We'd then have background task responsible for doing this polling after discovering a new rack subnet/sled-agent ip. Once all sled-agents start we'll allow the operator to go ahead and adopt the rack. If they don't all start eventually, we'll have a support issue. That bg task (or another one) can also attempt to send the completion event until success to the sled-agent hosting the join service when the operator selects adoption via the external API. It would then be responsible for publishing the bootstore generation and triggering the movement of these sleds into the sled table and the rack into the rack table. That should allow reconfigurator to take over.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, I like this plan. I would rather the unadopted sled/rack listing API be able to just query the table without having to chain in another separate API call to grab the sled-agent statuses. I'll go ahead and add the status to this table. Would you say it's a good idea to add a new
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure if you need an enum here, although I'm not against one. You could probably just use a bool @jgallagher What's your take?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah having thought some more about this, I guess since the database doesn't have algebraic types for enums, a bool flag and a nullable error string probably do make more sense. I was also sort of thinking we might want to future-proof ourselves in case we want to represent more fine-grained startup states at some point in the future, but if we do decide to do that then it'll be easy enough to write a migration to turn the bool into an enum anyway, so probably better to keep it simple for now. (Though happy to change my mind if @jgallagher responds with a different take.)
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Yeah, this is the way I'd go, I think. A wrinkle here is that adding a new enum in a migration is really easy, but changing an existing enum can be a real pain. Is
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
It is not relevant at that point. A constraint would be good!
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah I'd thought about a constraint, but then figured we might want to keep the last-seen error around, in case somebody wants to be able to see what it was for some reason, even if the error went away and the sled-agent is now started. Admittedly that is sort of a potentially confusing/weird state for the table row to be in, though, and I assume the error would still be in the logs somewhere if somebody needed to see what it was for debugging reasons, so I'll go ahead and change this. |
||
| ) | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -9730,6 +9730,51 @@ CREATE TABLE IF NOT EXISTS omicron.public.fm_config ( | |
| ) | ||
| ); | ||
|
|
||
| -- Tracks other racks that have joined the underlay network but haven't been | ||
| -- adopted into the multirack cluster. | ||
| -- | ||
| -- If/when a rack is adopted, its row in this table should be marked deleted, | ||
| -- and a corresponding row should be inserted into the `rack` table. | ||
| CREATE TABLE IF NOT EXISTS omicron.public.unadopted_rack ( | ||
| -- This should be the same id that gets used for the `rack` table, as well | ||
| -- as the `rack_id` column in `unadopted_sled`: | ||
| id UUID PRIMARY KEY, | ||
|
|
||
| time_created TIMESTAMPTZ NOT NULL, | ||
| time_modified TIMESTAMPTZ NOT NULL, | ||
| time_deleted TIMESTAMPTZ, | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's fine to defer this for now, but just a heads-up that when you start writing queries against this and want to filter out deleted rows, you'll probably nee a unique partial index (lots of other examples in here though).
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah cool, thanks for the pointer, I hadn't encountered/noticed that yet. Will add, either in this PR or the next one in the stack, but probably this one just to save the boilerplate of doing another migration later on. |
||
|
|
||
| -- The IPv6 underlay /56 prefix for the rack: | ||
| rack_subnet INET NOT NULL, | ||
|
|
||
| -- Socket address of the multirack join service running on a sled-agent in | ||
| -- the unadopted rack: | ||
| join_service_ip INET NOT NULL, | ||
| join_service_port INT4 | ||
| CHECK (join_service_port BETWEEN 0 AND 65535) | ||
| NOT NULL | ||
| ); | ||
|
|
||
| -- Tracks sleds from other racks that have joined the underlay network but | ||
| -- haven't been adopted into the multirack cluster. | ||
| -- | ||
| -- If/when these sleds are adopted, these rows should be marked deleted, and | ||
| -- there will subsequently be corresponding rows created in the `sled` and | ||
| -- `inv_sled_agent` tables via the normal mechanisms. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't believe that sled-agents will be added to the sled-table via any normal process. We'll need to put them there in the background task after receiving the completion response that sets the bootstore generation. I opened an issue for this earlier today.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I take it back. The sled-agent always upserts itself once it can locate a nexus. That might be fine. What I'm worried about is the sled getting added to the sled-table before we are ready - mainly before the bootstore generation for this rack is updated and the rack is present in the rack table.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah right, yeah, I remember originally we were thinking about having a separate parallel insertion path but I figured if we're not inserting I'm less familiar with the bootstore stuff, so not sure if that throws a wrench in things, but if we just needed to wait for a record to be inserted in the This is also getting pretty tangential to this PR though so we can probably discuss this later, maybe in the sync meeting tonight.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Or, well, I don't know. I published that comment and then immediately started questioning this. Maybe the initial inserts into
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it depends somewhat on how they find Nexus, which is DNS. Can they find DNS via the hardcoded addresses for the AZ? In any event you are right, we should discuss at huddle. I already added a note last night 😎 . |
||
| CREATE TABLE IF NOT EXISTS omicron.public.unadopted_sled ( | ||
| id UUID PRIMARY KEY, | ||
| rack_id UUID NOT NULL, | ||
|
|
||
| time_created TIMESTAMPTZ NOT NULL, | ||
| time_modified TIMESTAMPTZ NOT NULL, | ||
| time_deleted TIMESTAMPTZ, | ||
|
|
||
| -- Baseboard information about the sled | ||
| serial_number STRING(63) NOT NULL, | ||
| part_number STRING(63) NOT NULL, | ||
|
|
||
| ip INET NOT NULL | ||
| ); | ||
|
|
||
| -- Keep this at the end of file so that the database does not contain a version | ||
| -- until it is fully populated. | ||
|
|
@@ -9740,7 +9785,7 @@ INSERT INTO omicron.public.db_metadata ( | |
| version, | ||
| target_version | ||
| ) VALUES | ||
| (TRUE, NOW(), NOW(), '309.0.0', NULL) | ||
| (TRUE, NOW(), NOW(), '310.0.0', NULL) | ||
| ON CONFLICT DO NOTHING; | ||
|
|
||
| COMMIT; | ||
Uh oh!
There was an error while loading. Please reload this page.