AMD Zen4 7950X

[tofurky]

After running for a while, several cores will eventually get a bad reading of 429C and stick there until corefreqd is restarted.

Ryzen 7950X w/PBO enabled on ASUS ProArt X670E-Creator, kernel 6.6.32 built with gcc 13.2.0 on Ubuntu 24.04.

CoreFreq 117b8c0 from Wed Feb 28 02:40:18 2024 +0100. Currently now testing with tip of master, but the issue takes some time to show up and I'll update if/when it does.

System PBO offsets were tuned to undervolt most cores and overvolt 1 for stability. RAM is DDR5-6000 with Infinity Fabric at UCLK/1. UEFI is latest available with AGESA 1.1.7.0. No application crashes or other issues seen in dmesg. This issue has been happening as long as I've been letting corefreqd run continuously, so since last August according to .bash_history.

Unsure if this means there's an occasional bad read from the SMU or there's something else at play? Please let me know if you need any other information, thanks.

Screenshot:

[cyring]

Hello,

Thank you for reporting this issue.

Sorry for all details that will follow; I need to simulate the call flow of thermal which starts in this Driver function

CoreFreq/x86_64/corefreqk.c

Line 16233 in 7c8d354

static void CCD_AMD_Family_19h_Zen4_Temp(CORE_RO *Core)

Temperature is indeed gotten from SMU; TCCD_REGISTER address interlaced per CCD

CoreFreq/x86_64/amd_reg.h

Line 1504 in 7c8d354

} TCCD_REGISTER;

The SMU current temperature CurTmp is subtracted by 49 whenever CurTempRangeSel bit is set to one

Then thermal data are processed by the Daemon:

CoreFreq/x86_64/corefreqd.c

Line 1203 in 7c8d354

CFlip->Thermal.Sensor = RO(Core)->PowerThermal.Sensor;

and maths start here:

CoreFreq/x86_64/corefreqd.c

Line 316 in 7c8d354

COMPUTE_THERMAL(AMD_17h,

Temperature is finally computed by this macro:

CoreFreq/x86_64/coretypes.h

Line 709 in 7c8d354

#define COMPUTE_THERMAL_AMD_17h(Temp, Param, Sensor) \

---

Question: do you have any other thermal agent running in parallel of CoreFreq ?
For instance, the kernel module k10temp ?

[tofurky]

Question: do you have any other thermal agent running in parallel of CoreFreq ? For instance, the kernel module k10temp ?

Yep, k10temp is loaded and polled every 5m by munin-node and much more frequently by MATE sensors applet (Tctl only). Possibly there's 2 calls at once sometimes? There have never been super high values reported by k10temp by munin, though.

k10temp-pci-00c3
Adapter: PCI adapter
Tctl:         +51.5°C  
Tccd1:        +39.4°C  
Tccd2:        +38.8°C  

The graph tends to squash any one time high readings, but each point below represents 15m (3x 5m readings) I think. System uptime from the previous screenshot showing 429C is 18 days uptime.

[cyring]

There's an access conflict to the SMU between corefreqk.ko and k10temp which has to be unloaded.

Fyi you can quickly reproduce this conflict by lowering the collect interval to 100ms in the corefreq-cli

[tofurky]

I haven't been able to reproduce it yet doing the following:

• Running the following to poll k10temp Tctl, Tccd1, Tccd2 as quickly as possible:
  while true; do cat /sys/class/hwmon/hwmon4/temp{1,3,4}_input ; done
• Running corefreq-cli at 100ms interval

Been running for about 5m now, I'll let it run longer and see if I see the high temps.

Edit: after 45m, one core went to 429C. I'll continue testing without k10temp loaded.

[cyring]

I haven't been able to reproduce it yet doing the following:

• Running the following to poll k10temp Tctl, Tccd1, Tccd2 as quickly as possible:
  while true; do cat /sys/class/hwmon/hwmon4/temp{1,3,4}_input ; done
• Running corefreq-cli at 100ms interval

Been running for about 5m now, I'll let it run longer and see if I see the high temps.

Edit: after 45m, one core went to 429C. I'll continue testing without k10temp loaded.

I did not noticed your Issue Edit.

If you confirm no other SMU agent is running in parallel, it may be possible it is visible since Raphael.

As a day to day, I'm programming on AMD Matisse with particular kernel boot parameters documented in Wiki
Once a while, before releasing especially, I'm doing full non regression tests with most kernel modules loaded but k10temp and occasionally amd_pstate

Now rebooting in full modules loaded ...

[tofurky]

Ah, sorry about the edit, yeah I didn't want to generate too many e-mails.

I've left it running at 100ms overnight and one core has gone to 429C. k10temp isn't loaded.

matt@aquos:~$ lsmod |grep k10temp
matt@aquos:~$ 

[tofurky]

I'm using amd_pstate=active for CPU frequency control; I imagine this might interact with the SMU in some way? It is available in kernel 6.5+ I believe.

[tofurky]

Sorry, make that 2 cores. Needed to scroll down. I was thinking it was a bit odd that if there's 16 physical cores there'd be 32 separate temperature readings.

024 in the screenshot is CCD 1, ID 8 and shows 429C, but 008 is also CCD 1, ID 8 and doesn't show the high reading. Maybe this is just a quirk of how the temperature data is retrieved? Seems like they both should be at 429C though, if it's the same core. I guess CCD_AMD_Family_19h_Zen4_Temp() mentioned above doesn't take topology into account and lets the SMU return a separate temperature for each "core" even if it's just a thread versus physical core?

[cyring]

I have a 100ms session running on my Ryzen 9 3950X and issue has not appeared yet.

But looking at your screenshots, can we say that 429 is only displayed in the Max TMP column ?

[tofurky]

Yes, it's only shown in the max column. I see 429C in max columns for cores 1, 3, 7, 11, 13, 16, 24, 26 now.

[cyring]

I'm using amd_pstate=active for CPU frequency control; I imagine this might interact with the SMU in some way? It is available in kernel 6.5+ I believe.

Depending on the firmware implementation amd-pstate.c is requesting either the ACPI CPPC either MSR registers (similar to Intel HWP)
I have not read SMU function references in that driver but it may indirectly.

[cyring]

Yes, it's only shown in the max column. I see 429C in max columns for cores 1, 3, 7, 11, 13, 16, 24, 26 now.

Great if I can say; it's narrowing the investigation.
It could be a math side effect in the computing macros.

CoreFreq/x86_64/coretypes.h

Line 631 in 7c8d354

/* Sensors formulas and definitions.

[tofurky]

Well, I haven't been watching it for hours so it's possible that it appears once in the TMP column before appearing in the Max column.

But, it's always 429C. Never in between e.g. 300C. There's not wrong MIN values, either.

I am running 'corefreq-cli -C > corefreq.out' so I can check if the instantaneous values show 429C.

[cyring]

Maybe this is just a quirk of how the temperature data is retrieved?

In fact, sensors are retrieved depending on the sensor scope.
In the UI Settings window you can change the thermal scope among 4 possibilities: None, Package, Core or SMT (Thread)

[cyring]

Tip: at any time press the star key * to reset values, including the Min and Max of each CPU sensors, frequencies and so on.