Add metrics representing project quota utilization. by jmcarp · Pull Request #11586 · oxidecomputer/omicron

jmcarp · 2026-10-23T21:11:55Z

Analogous to #11500, but covering projects rather than silos. Project have utilization but not capacity, so we only emit utilization metrics.

Note: to parallel the existing DataStore::silo_utilization_list, and the silo_utilization view that it queries, we add a
DataStore::project_utilization_list method and project_utilization view. We don't add a /v1/system/utilization/projects endpoint to parallel /v1/system/utilization/silos, but we could in a future patch if useful.

Part of oxidecomputer/customer-support#1383.

jmcarp · 2026-10-23T21:15:15Z

        .map_err(|e| public_error_from_diesel(e, ErrorHandler::Server))
    }
+
+    pub async fn project_utilization_list(


This is meant to parallel silo_utilization_list. Arguably it's too parallel, though--we introduce project_quota view to match the existing silo_quota view, which doesn't (yet?) have any other consumers. If this is too much boilerplate, we can skip the new view and just write the equivalent sql with diesel here.

bnaecker

Looks pretty good! Some small questions and suggestions.

bnaecker · 2026-10-23T21:40:13Z

+                        now,
+                        &target,
+                        &project_quota::CpuProvisioned {
+                            datum: u64::try_from(util.cpus_provisioned)?,


I might recommend we log this error to convert to a u64, but that we don't bail. That way we get whatever data we can, even if this conversion fails.

Reasonable, done.

bnaecker · 2026-10-23T21:41:12Z

+    s.cpus_allocated AS silo_cpus_allocated,
+    s.memory_allocated AS silo_memory_allocated,
+    s.storage_allocated AS silo_storage_allocated,


Do we need these columns? I don't see the silo allocations used when emitting the metrics.

I'm dropping these columns from the view. Like you wrote below, we only use them for filtering, so I pushed that logic into the view and out of diesel.

bnaecker · 2026-10-23T21:42:24Z

+    c.cpus_provisioned AS cpus_provisioned,
+    c.ram_provisioned AS memory_provisioned,
+    c.virtual_disk_bytes_provisioned AS storage_provisioned,
+    s.silo_discoverable as silo_discoverable


IIUC, we're filtering on this being true in the method that selects these records from the database. Do we need them, in that case? Or could we have a WHERE clause here that only selects those silos which are discoverable?

Yep, added the WHERE here.

smklein · 2026-10-23T21:29:54Z

+    pub project_id: Uuid,
+    pub project_name: Name,
+    pub silo_id: Uuid,


ugh, we actually do have a DbTypedUuid wrapper for db-based strongly-typed UUIDs, but I'm not sure we have typed uuids for projects or silos yet.

I'm digging into this in #11593

Neat, I'll update here once that lands.

Updated to use ProjectId rather than Uuid.

smklein · 2026-10-23T21:35:42Z

            }
        }

+        let mut project_paginator =


wayyyy up on line 76 you declared the vec samples, which you're sharing for silos and projects.

I don't really have a beef with this re-use, but this structure means "an error with either side prevents any samples from being collected". Is this desired? I understand that it "highlights errors" more quickly, but it basically makes metrics collection sensitive to single-point-of-failure

smklein · 2026-10-23T21:40:15Z

This is now out-of-date. I dunno if you want to add projects here, or reference a list somewhere else

Updated to be more generic, and pointing to toml files for metric definitions rather than restating here.

smklein · 2026-10-23T21:41:14Z

+/**
+ * A view of the amount of provisioned resources on a given project.
+ */
+CREATE VIEW IF NOT EXISTS omicron.public.project_utilization


nitpick, should this exist a little closer to projects / silos?

smklein · 2026-10-23T21:55:03Z

+                    .or(dsl::silo_cpus_allocated.gt(0))
+                    .or(dsl::silo_memory_allocated.gt(0))
+                    .or(dsl::silo_storage_allocated.gt(0)),
+            )


Could we do something like:

/// Silos whose utilization we report. fn silo_utilization_reported() -> impl Expression<...> { use nexus_db_schema::schema::silo_utilization::dsl; dsl::silo_discoverable .eq(true) .or(dsl::cpus_allocated.gt(0)) .or(dsl::memory_allocated.gt(0)) .or(dsl::storage_allocated.gt(0)) .and(dsl::silo_id.ne(DEFAULT_SILO_ID)) }

To share this query with the silo version?

We could have, but I took @bnaecker's suggestion to do this filtering in the crdb view rather than DataStore, so now we don't need to repeat this filter for project_utilization_list at all.

smklein · 2026-10-23T21:57:38Z


+#[derive(Queryable, Debug, Clone, Selectable, Serialize, Deserialize)]
+#[diesel(table_name = project_utilization)]
+pub struct ProjectUtilization {


From what I can tell, we're never using the project_name or silo_name in this PR. Are we planning on using them?

Oh, right, those are left over from the SiloUtilization struct I started from, and would only be used if we wanted a /v1/system/utilization/projects endpoint to match /v1/system/utilization/silos. I was thinking that would be worth adding for symmetry, but I can drop the columns here for now, and add them back if/when we add that endpoint.

Analogous to #11500, but covering projects rather than silos. Project have utilization but not capacity, so we only emit utilization metrics. Note: to parallel the existing `DataStore::silo_utilization_list`, and the `silo_utilization` view that it queries, we add a `DataStore::project_utilization_list` method and `project_utilization` view. We don't add a `/v1/system/utilization/projects` endpoint to parallel `/v1/system/utilization/silos`, but we could in a future patch if useful. Part of oxidecomputer/customer-support#1383.

* Drop unused project_utilization fields, push filtering to view. * Return separate silo/project quota counts. * Clean up error handling.

jmcarp · 2026-11-03T17:42:29Z

This is ready for another look from @smklein and/or @bnaecker, time permitting.

jmcarp requested review from bnaecker and smklein October 23, 2026 21:11

jmcarp commented Oct 23, 2026

View reviewed changes

bnaecker reviewed Oct 23, 2026

View reviewed changes

smklein reviewed Oct 23, 2026

View reviewed changes

jmcarp force-pushed the jmcarp/project-capacity-metrics branch 4 times, most recently from 1093cf4 to 2b2b73b Compare October 28, 2026 14:57

jmcarp added 4 commits October 29, 2026 13:24

Project quota metrics: review cleanup.

e4ec6c1

* Drop unused project_utilization fields, push filtering to view. * Return separate silo/project quota counts. * Clean up error handling.

Project quota metrics: switch from Uuid to ProjectId.

6fb40df

Project quota metrics: switch silo IDs from Uuid to SiloId.

bc7efb5

jmcarp force-pushed the jmcarp/project-capacity-metrics branch from 2b2b73b to bc7efb5 Compare October 29, 2026 19:23

Conversation

jmcarp commented Oct 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bnaecker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmcarp commented Nov 3, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants