Debug CI failures for latest openvox-agent builds#312
Closed
bastelfreak wants to merge 6 commits intomainfrom
Closed
Debug CI failures for latest openvox-agent builds#312bastelfreak wants to merge 6 commits intomainfrom
bastelfreak wants to merge 6 commits intomainfrom
Conversation
Contributor
Author
|
I added some debug statements in 36247f9. That should allow us to reproduce it locally with the correct environment parameters. The package builds happens in https://github.com/OpenVoxProject/openvox/actions/runs/21750529041. And the new acceptance test pipeline runs in https://github.com/OpenVoxProject/openvox/actions/runs/21750761522/job/62747866956 |
Contributor
Author
|
from the latest run: |
Contributor
Author
|
AHA! Not that this helps me a lot right now, but it at least enables us to reproduce it locally. |
Contributor
Author
|
A bit more verbose: |
Contributor
Author
|
The different snapshot builds with rpm/deb packages are available at https://artifacts.voxpupuli.org/openvox-agent/. |
We assume that the ruby 3.2.9->3.2.10 upgrade in puppet-runtime 2026.01.30.1 broke our builds.
Contributor
Author
|
@sebastianrakel and I did some debugging. We downgraded puppet-runtime in 0f493b1: diff --git a/packaging/configs/components/puppet-runtime.json b/packaging/configs/components/puppet-runtime.json
index 27d523c6de..7210090a27 100644
--- a/packaging/configs/components/puppet-runtime.json
+++ b/packaging/configs/components/puppet-runtime.json
@@ -1 +1 @@
-{"location":"https://s3.osuosl.org/openvox-artifacts/puppet-runtime/2026.01.30.1/","version":"2026.01.30.1"}
+{"location":"https://s3.osuosl.org/openvox-artifacts/puppet-runtime/2026.01.29.2/","version":"2026.01.29.2"}This downgrades Ruby 3.2.10 to the last known working version, 3.2.9: diff --git a/configs/components/curl.rb b/configs/components/curl.rb
index 5c6e2d4..da18188 100644
--- a/configs/components/curl.rb
+++ b/configs/components/curl.rb
@@ -2,8 +2,8 @@
# Component release information: https://github.com/curl/curl/releases
#####
component 'curl' do |pkg, settings, platform|
- pkg.version '8.18.0'
- pkg.sha256sum 'e9274a5f8ab5271c0e0e6762d2fce194d5f98acc568e4ce816845b2dcc0cf88f'
+ pkg.version '8.17.0'
+ pkg.sha256sum 'e8e74cdeefe5fb78b3ae6e90cd542babf788fa9480029cfcee6fd9ced42b7910'
pkg.url "https://curl.se/download/curl-#{pkg.get_version}.tar.gz"
pkg.mirror "#{settings[:buildsources_url]}/curl-#{pkg.get_version}.tar.gz"
diff --git a/configs/components/dmidecode.rb b/configs/components/dmidecode.rb
index 0556828..c55c1c2 100644
--- a/configs/components/dmidecode.rb
+++ b/configs/components/dmidecode.rb
@@ -2,11 +2,11 @@
# Component release information: https://github.com/mirror/dmidecode/tags
#####
component 'dmidecode' do |pkg, settings, platform|
- pkg.version '3.7'
- pkg.sha256sum '2c3aed12c85a1e6a9410d406d5e417c455466dc1bc7c89278bb32cf7cad91e8a'
+ pkg.version '3.6'
+ pkg.sha256sum 'e40c65f3ec3dafe31ad8349a4ef1a97122d38f65004ed66575e1a8d575dd8bae'
pkg.apply_patch 'resources/patches/dmidecode/dmidecode-install-to-bin.patch'
- pkg.url "https://download-mirror.savannah.gnu.org/releases/dmidecode/dmidecode-#{pkg.get_version}.tar.xz"
+ pkg.url "http://download-mirror.savannah.gnu.org/releases/dmidecode/dmidecode-#{pkg.get_version}.tar.xz"
pkg.mirror "#{settings[:buildsources_url]}/dmidecode-#{pkg.get_version}.tar.xz"
pkg.environment 'LDFLAGS', settings[:ldflags]
diff --git a/configs/components/openssl-3.0.rb b/configs/components/openssl-3.0.rb
index e3dd619..56111e3 100644
--- a/configs/components/openssl-3.0.rb
+++ b/configs/components/openssl-3.0.rb
@@ -6,8 +6,8 @@
# need to move to the 3.5.x LTS stream in the next year.
#####
component 'openssl' do |pkg, settings, platform|
- pkg.version '3.0.19'
- pkg.sha256sum 'fa5a4143b8aae18be53ef2f3caf29a2e0747430b8bc74d32d88335b94ab63072'
+ pkg.version '3.0.18'
+ pkg.sha256sum 'd80c34f5cf902dccf1f1b5df5ebb86d0392e37049e5d73df1b3abae72e4ffe8b'
pkg.url "https://github.com/openssl/openssl/releases/download/openssl-#{pkg.get_version}/openssl-#{pkg.get_version}.tar.gz"
pkg.mirror "#{settings[:buildsources_url]}/openssl-#{pkg.get_version}.tar.gz"
diff --git a/configs/components/ruby-3.2.rb b/configs/components/ruby-3.2.rb
index afe4682..273db78 100644
--- a/configs/components/ruby-3.2.rb
+++ b/configs/components/ruby-3.2.rb
@@ -6,8 +6,8 @@
# The file name of the ruby component must match the ruby_version
#####
component 'ruby-3.2' do |pkg, settings, platform|
- pkg.version '3.2.10'
- pkg.sha256sum '880acb05e08da8c559c56a13e512bae1b472da67c72ebb750c765f9c2134e689'
+ pkg.version '3.2.9'
+ pkg.sha256sum 'abbad98db9aeb152773b0d35868e50003b8c467f3d06152577c4dfed9d88ed2a'
ruby_dir = settings[:ruby_dir]
ruby_bindir = settings[:ruby_bindir]But the acceptance test still fails in https://github.com/OpenVoxProject/openvox/actions/runs/21754287120/job/62760179025 |
Contributor
Author
|
from puppet-runtime: this seems related to our issue |
Contributor
Author
|
We rebuild puppet-runtime and removed the uri gem. We explicitly managed it in 03fb02fd0bcdcca5224c4c98d6086afda35ea0bc: I assume the rm in 047030607d7c768825de674b0c368f5043af0c3a / OpenVoxProject/puppet-runtime@0470306 breaks the uri gem if it exists twice (because it's also a std gem). We fixed this in OpenVoxProject/puppet-runtime#111. the new tag 2026.02.06.1 contains that change. I restarted the CI for the acceptance2 branch and the previously failing test passes now: https://github.com/OpenVoxProject/openvox/actions/runs/21763584495 I'm now promoting puppet-runtime 2026.02.06.1 to main: #313. Afterwards we can build new packages and run the full acceptance test pipeline again. I hope this roughly explains is issue. tl;dr: puppet-runtime doesn't handle updated std gems very well (https://stdgems.org/). ToDo: Update configs/components/_base-rubygem.rb in puppet-runtime to warn/block updates to stdgems or delete the old version properly (also talk to @rwaffen, because he has the same challenge in voxbox) |
Contributor
Author
|
proof that it works now: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
latest builds of openvox-agent have failing pipelines for the acceptance tests. This PR is for debugging purposes only. I removed some tests to speedup the overall execution time. We see the following error on all platforms:
The github action for acceptance tests is triggered manually: https://github.com/OpenVoxProject/openvox/actions/workflows/acceptance.yml
start options:
The failing test: https://github.com/OpenVoxProject/openvox/blob/acceptance2/acceptance/tests/provider/package/puppetserver_gem.rb
tests passed in the past: https://github.com/OpenVoxProject/openvox/actions/runs/17598424682 (last successful run)
first failing run: https://github.com/OpenVoxProject/openvox/actions/runs/21496664654
The tests use actual rpm/deb packages, which need to be build first in another pipeline (in case someone adds code, then you also need to adjust OpenVox Agent Version for the acceptance.yml pipeline): https://github.com/OpenVoxProject/openvox/actions/workflows/build.yml
the failing code: https://github.com/OpenVoxProject/openvox/blob/acceptance2/lib/puppet/provider/package/puppetserver_gem.rb#L143-L173