[TW#20534] Light sleep stalls (blocks) at random intervals and is disabling watchdogs

[Electrozest]

Calling esp_light_sleep_start() with less than one second delay set, and calling the function every second, the esp_light_sleep_start() function stalls (blocks) after 1-3 hours. It then remains blocked until a manual reset.

My code is enabling the hardware WDT at startup so I would have expected a WDT reset following the blocking condition, however the hardware WDT was failing to act. On inspection of the code I realise that unexpectedly the esp_light_sleep_start() is enabling and disabling the hardware WDT. Unfortunately on exit the function leaves the hardware watchdog timer disabled!!

There are two critical issues:
i) esp_light_sleep_start() blocks at random
ii) esp_light_sleep_start() is disabling the hardware WDT

The esp_light_sleep_start() function is found in esp-idf/components/esp32/sleep_modes.c

Until these issues are resolved the ESP32 is not suitable for remote operation if light sleep mode is required.

This issue is also related to: espressif/arduino-esp32#908

[igrr]

Hi @Electrozest, we have fixes for light sleep blocking at random in review at the moment. Should be merged in the next few days.

Regarding hardware WDT, light sleep uses RTC watchdog. RTC watchdog is not supposed to be used by applications directly, please use Timer Group watchdogs (which are described in the TRM).

[Electrozest]

Hi @igrr, thank you. I look forwards to testing the new fixes.

My application uses FreeRTOS task watchdog with esp_task_wdt_feed() and a software watchdog implemented with interrupt timer, however both of these are not triggering whenever the esp_light_sleep_start() function blocks, so system remains frozen after esp_light_sleep_start() failure.

I assume once esp_light_sleep_start() is fixed, the task and software watchdogs will function correctly, unless I need to also configure light sleep to wake on additional interrupts related to task and/or interrupt timer watchdogs?

The TRM mentions Main System Watchdog Timer (MWDT) and RTC Watchdog (RWDT). I wondered if FreeRTOS already be using the MWDT for task watchdog purposes?

[igrr]

configure light sleep to wake on additional interrupts related to task and/or interrupt timer watchdogs

It is not possible to wake up from light sleep based on peripheral interrupts, since the peripherals are clock gated in light sleep. The only wakeup methods are the ones in RTC domain, and are listed in IDF sleep modes documentation.

FreeRTOS uses Timer Group 0 and Timer Group 1 watchdogs for task and interrupt watchdog functionality.

[Electrozest]

Following the commit for this issue, I am observing a new issue.

Light sleep on its own is observed to function without stalling.

However any light sleep operations after a single deep sleep operation (and subsequent restart) are stalling forever.

[igrr]

@Electrozest Failure to exit light sleep after going through deep sleep has been fixed in ac623a9. Please update to latest master and check. If you are still seeing the issue, please share the code snippet to reproduce and your sdkconfig file. For reference, our test case for light sleep and deep sleep is here:

esp-idf/components/esp32/test/test_sleep.c

Lines 47 to 52 in 92c469b

	TEST_CASE("light sleep followed by deep sleep", "[deepsleep][reset=DEEPSLEEP_RESET]")
	{
	esp_sleep_enable_timer_wakeup(1000000);
	esp_light_sleep_start();
	esp_deep_sleep_start();
	}

. Running it multiple times (i.e. light sleep - deep sleep - light sleep - deep sleep - ...) seems to work.

[Electrozest]

@igrr As requested, I have updated my esp-idf to latest master and recompiled. The low power sleep is still failing after deep sleep. The deep sleep is functioning for correct time period.

Note that the first low power sleep after deep sleep is ok. It is the subsequent low power sleeps that lock.

So a sequence such as:
light sleep - light sleep - light sleep - light sleep - deep sleep - light sleep - light sleep - light sleep

[igrr]

Just to check, when you say "low power sleep", do you mean light sleep?

[Electrozest]

Yes.

[igrr]

I've run the following test, which worked (for a minute or so):

#include "esp_sleep.h"

void app_main()
{
    esp_sleep_enable_timer_wakeup(100000);

    esp_light_sleep_start();
    esp_light_sleep_start();
    esp_light_sleep_start();
    esp_light_sleep_start();
    esp_light_sleep_start();

    esp_deep_sleep_start();
}

Is this representative of the case you are testing?

[Electrozest]

Body of app_main is similar to below.

  esp_sleep_enable_timer_wakeup(1000000);
  esp_light_sleep_start();
  esp_light_sleep_start();
  esp_light_sleep_start();
  esp_light_sleep_start();
  esp_light_sleep_start();
  esp_sleep_enable_timer_wakeup(5000000);
  esp_deep_sleep_start();

I notice now that I have updated to latest ESP-IDF master and recompiled that my wifi client code sends TCPIP, but my PC server no longer receives any packets. Identical hardware unit with same WiFi code prior to ESP-IDF update from this morning still transmits packet. Looks like I need to revert to older version of ESP-IDF as my application requires WiFi and TCPIP.

[Electrozest]

Confirmed, I have tried switching to/from esp-idf master and esp-idf from mid April and WiFiClient TCPIP using client.connect() and client.write() no longer sends packets even though there are no error messages from the tcpip client, but works correctly with esp-idf from mid April.

[Electrozest]

With regards to the light sleep and deep sleep.
After boot, I switch from internal RTC clock to external 32.768kHz XTAL CLK. Switching to external 32.768k XTAL causes 1 second sleep to become 5+ seconds sleep, when the delay should be almost the same.

[igrr]

After boot, I switch from internal RTC clock to external 32.768kHz XTAL CLK

Sorry, at the moment there is no supported way in ESP-IDF to do that. You need to decide whether 32k XTAL is used or not at build time.

Also I'm not sure how this fact is related to the issue being discussed. If you don't switch to 32k XTAL at run time, do you observe the issue (board locks up after light sleep)?

Can you please confirm that when you run exactly the following code, you see that the board locks up after light sleep?

#include "esp_sleep.h"
void app_main() {
  esp_sleep_enable_timer_wakeup(1000000);
  esp_light_sleep_start();
  esp_light_sleep_start();
  esp_light_sleep_start();
  esp_light_sleep_start();
  esp_light_sleep_start();
  esp_sleep_enable_timer_wakeup(5000000);
  esp_deep_sleep_start();
}

If this exact code does not reproduce the issue on your side, we need to converge on some code which does, and which you can share.