Repository navigation
Description
Consider the follow oxql query:
> oxide --profile r3 experimental system timeseries query --query 'get hardware_component:temperature | filter component_id == "PSU0" && timestamp > @now() - 1m | last 3'
{
"tables": [
{
"name": "hardware_component:temperature",
"timeseries": [
{
"fields": {
"chassis_model": {
"type": "string",
"value": "913-0000003"
},
"component_kind": {
"type": "string",
"value": "mwocp68"
},
"chassis_serial": {
"type": "string",
"value": "BRM45220018"
},
"chassis_revision": {
"type": "u32",
"value": 8
},
"description": {
"type": "string",
"value": "PSU 0 MCU"
},
"hubris_archive_id": {
"type": "string",
"value": "a7cec194aa73a90f"
},
"chassis_kind": {
"type": "string",
"value": "power"
},
"rack_id": {
"type": "uuid",
"value": "48759651-d754-43e2-ad49-64dde3adc1c2"
},
"sensor": {
"type": "string",
"value": "psu0mcu"
},
"gateway_id": {
"type": "uuid",
"value": "dfeef10e-0dda-432d-a06a-f103456881ce"
},
"component_id": {
"type": "string",
"value": "PSU0"
},
"slot": {
"type": "u32",
"value": 0
}
},
"points": {
"timestamps": [
"2026-08-11T16:23:50.684568706Z",
"2026-08-11T16:23:50.684583584Z",
"2026-08-11T16:23:50.684595417Z"
],
"values": [
{
"metric_type": "gauge",
"values": {
"type": "double",
"values": [
15.75,
45.25,
40.75
]
}
}
]
}
},
{
"fields": {
"hubris_archive_id": {
"type": "string",
"value": "a7cec194aa73a90f"
},
"chassis_revision": {
"type": "u32",
"value": 8
},
"chassis_serial": {
"type": "string",
"value": "BRM45220018"
},
"sensor": {
"type": "string",
"value": "psu0mcu"
},
"component_kind": {
"type": "string",
"value": "mwocp68"
},
"component_id": {
"type": "string",
"value": "PSU0"
},
"rack_id": {
"type": "uuid",
"value": "48759651-d754-43e2-ad49-64dde3adc1c2"
},
"chassis_model": {
"type": "string",
"value": "913-0000003"
},
"description": {
"type": "string",
"value": "PSU 0 MCU"
},
"chassis_kind": {
"type": "string",
"value": "power"
},
"gateway_id": {
"type": "uuid",
"value": "7cff00d5-b96b-4f9f-b6dd-cb8f8d5e42ba"
},
"slot": {
"type": "u32",
"value": 0
}
},
"points": {
"timestamps": [
"2026-08-11T16:23:49.683031412Z",
"2026-08-11T16:23:49.683044597Z",
"2026-08-11T16:23:49.683053594Z"
],
"values": [
{
"metric_type": "gauge",
"values": {
"type": "double",
"values": [
15.875,
45.25,
40.75
]
}
}
]
}
}
]
}
]
}
In particular, the timestamps of the last three values:
"timestamps": [
"2026-08-11T16:23:50.684568706Z",
"2026-08-11T16:23:50.684583584Z",
"2026-08-11T16:23:50.684595417Z"
],
These timestamps are the same down to the millisecond. But we sample the sensor every 1s. Why are we getting multiple almost-identical timestamps for a single sensor, with very different values?
This seems to be happening because the psu0mcu sensor on the mwocp68 actually includes multiple sensors, and we don't distinguish among them in hubris. If the different sensors measured the same thing, this might not matter much, but I believe they're actually measuring different things (ambient vs hot-spot temperatures). So I think these metrics are basically uninterpretable until we include the full sensor names.
It looks like we would fix this by updating metadata around https://github.com/oxidecomputer/hubris/blob/9e553c76105497896a725b89558b46fda0ee10c6/app/psc/base.toml. It would also be helpful to include some kind of check to avoid duplicate sensor names if possible, because we also have a similar conflict in hardware_component:fan_speed, also for the mwocp68.
More selfishly, I noticed this because I tried to ingest these metrics into prometheus, which only has millisecond timestamp precision, and hit errors due to conflicting timestamps. That isn't necessarily an Oxide bug, but I think the missing sensor information actually is a bug.
Possibly relevant to your interests @hawkw @evan-oxide.
Activity
hawkw commentedon Aug 11, 2026
Yeah, I think you're right that we need to add something in Hubris to disambiguate both these and the fan speed sensors (which, IIRC, are for the fans on the front and back of the PSU?). We should probably move this issue to the Hubris repo, since I believe the fix for this is going to be entirely there.
hawkw commentedon Aug 11, 2026
We can probably do something about that in the Hubris build system, good idea!
hawkw commentedon Aug 12, 2026
Looking closer at the Hubris build system, we do currently have a way to assign names to multiple sensors of the same type on a given device. See, for example, the Cosmo fan controller:
hubris/app/cosmo/base.toml
Lines 1148 to 1150 in 1a6600c
However, the way this works currently is pretty dumb: when generating sensor descriptions, a sensor's name is determined as follows:
Here's the code that implements this:
hubris/build/i2c/src/lib.rs
Lines 382 to 403 in 1a6600c
Note
The way this works is both confusing and remarkably under-documented. We really need to add some better documentation explaining how sensor names are (or are not) selected.
The unfortunate thing about this code is that the names are determined based on the order that the sensors are added. In particular, sensors are added in order based on the type, here:
hubris/build/i2c/src/lib.rs
Lines 329 to 359 in 1a6600c
I think this includes the indices generated for the various PMBus sensors, even though they don't actually hit the
nameslist. So there will need to be dummy entries in that list for the PMbus sensors so that it's the correct length.The code in
task-sensor-pollingthat reads the temperature and fan speed sensors is just selecting the PMbus page for each such sensor by index and posting to the corresponding sensor ID:hubris/task/sensor-polling/src/main.rs
Lines 109 to 131 in 9e553c7
So, we'd need to:
build_i2c, determine which index into the number of sensors added for the PSU corresponds to each PMBus page index used when polling sensorsAlternatively, we could change the way sensor name lists are specified to make the way they are selected not be quite so confusing. It seems like if we changed the I2C config file so that sensor name lists are specified per sensor type, it would be much simpler to add the correct names here. I don't even think that would break existing code too badly, since at a glance, the main place we're using the sensor name lists is for fan speed controllers, which only have one sensor type. That's probably why no one ever really noticed that using sensor name lists on devices that have multiple types of sensors is a nightmare.
build-i2cshould probably not permit multiple sensors of the same type to have the same name #2637jamesmunns commentedon Aug 21, 2026
I added the following check to
I2cSensorsDescription::new():This gives me the following warning for
psc-c(lightly edited for readability):fix MGS sensor metrics test