Skip to content

USHIFT-5296: Stabilize auto-recovery-extra test cases for QE env #4460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 27, 2025
Merged

USHIFT-5296: Stabilize auto-recovery-extra test cases for QE env #4460

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions test/assets/auto-recovery/10-auto-recovery.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,12 @@
[Unit]
OnFailure=microshift-auto-recovery.service
# Increase StartLimitInterval from default 10s to 25s to accomodate slower systems.
# Systemd will mark service as failed when it restarts StartLimitBurst amount
# (default is 5) within specified StartLimitInterval.
# In CI env, MicroShift restarts within ~1.3 seconds from a start, so 5 restarts
# fit in 10 seconds.
# In QE env, MicroShift needed around ~2.5 seconds to restart, so the
# StartLimitBurst (5) wasn't reached within 10 seconds. That resulted in systemd
# never marking microshift.service as failed (it kept restarting several hundred times)
# and the OnFailure= service not getting triggered.
StartLimitIntervalSec=25s
4 changes: 4 additions & 0 deletions test/assets/auto-recovery/red-script.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,9 @@ else
fi

echo "System is unhealthy and greenboot's 'deployment testing' procedure is not active - running auto-recovery for MicroShift"

echo "Making sure MicroShift is stopped and doesn't restart preventing from restoring the backup."
systemctl stop microshift

microshift restore --auto-recovery /var/lib/microshift-auto-recovery
systemctl reboot