Skip to content

Conversation

@jamescowens
Copy link
Member

@jamescowens jamescowens commented Nov 18, 2025

This PR updates critical components of the depends environment to Bitcoin Core 30.0, including the updating of depends Qt to Qt6 (currently at 6.7.3, the same as Bitcoin Core). Note that it also makes the necessary changes to the depends system to emit toolchain.cmake, and overhauls the cmake build system to be completely ready for the pending removal of the autotools build system. Currently Qt5 is still supported in a local build (for local use or packaging) via cmake on Linux, but this will be removed in the future once Qt5 is sunsetted.

Key Changes:

  • Qt6 Compatibility Upgrades the depends system to use Qt6 and
    align closer with the current depends in Bitcoin Core v30.0.
  • Build System Unification: The CMake configuration now supports
    Linux Native, Linux Static (depends), Windows Cross-Compile, and macOS Native
    targets with a single consistent workflow.
  • Test Framework Fixes: Resolved linking issues with Boost Test
    by conditionally applying dynamic/static linkage macros based on
    the target. Tests now build and pass on all platforms.
  • Static Linking: Refined src/test/CMakeLists.txt and dependencies
    to correctly handle transitive dependencies for static builds.
  • Depends: Updated boost.mk to ensure the Unit Test Framework
    library is compiled and installed correctly.
  • **Overhaul of CI/CD ** Complete overhaul of CI/CD, integrating CD into CI as envisioned by ci, cd: merge #2689.
  • ** Implementation of several helper scripts:**
    • build_targets.sh: A relatively friendly script that helps users do local compiles for three major targets, Linux Native, Linux Depends, and Windows Cross Compile (depends).
    • install_dependencies.sh: A helper script that works with build_targets.sh to assist in checking/installing dependencies for the major distributions: Debian, Ubuntu, Mint, Fedora/RHEL, OpenSUSE/SLES, and Arch/Manjaro. Note that Ubuntu, Fedora, and OpenSUSE (Leap and Tumbleweed) have been pretty thoroughly tested already.
    • contrib/devtools/run-local-ci.sh: A wrapper script around ACT primarily for executing CI jobs locally in docker containers, with concurrency limits. (Does NOT support native macOS jobs, obviously.)
    • contrib/devtools/split-logs.py: A short python program to split a combined output log from a run-local-ci.sh where several jobs are run in parallel.

CI/CD & Build System Updates (Summary)

As part of the CMake migration, the CI pipelines have been overhauled to be hermetic, reproducible, and developer-friendly.

1. Clear separation of purpose with CI configuration files

  • There are now four separate Github Actions configuration files:
    • cmake_quality.yml for quality control. There are two jobs: Sanitizers and the Linter.
    • cmake_production.yml for integrated CI and CD jobs that will produce artifacts and publish to releases. These currently consist of three jobs: Linux Static (Depends), Windows Cross Compile, and macOS Native (Homebrew Qt6). Each of these produces an artifact that will be downloadable on each CI invocation and published to tags (releases).
    • cmake_compatibility.yml for architecture compatibility checking. These currently consist of three jobs: Linux Native (System Qt6), Linux ARM64 (System Qt6), and Linux ARMHF (System Qt6). This checks compatibility with Qt6 and also uncovers architectural and word-size problems (i.e. Intel vs ARM and 64 vs 32 bit).
    • cmake_distros.yml for distro compatibility checking. These currently consist of six jobs: Latest Fedora (42), OpenSUSE Lep 15.6, OpenSUSE Tumbleweed, Arch Linux, Debian Unstable (Sid), and Linux Mint. This covers all three major packaging schemes, rpm, apt, and Packman, and covers some bleeding edge versus conservative distros. This serves as validation of the user build_targets.sh and install_dependencies.sh, since these scripts are used in this CI job set.
  • All jobs other than the Sanitizer and Linter are full builds with GUI. Qt6 is used on all CI jobs. The test suite is run on both core and GUI components. This includes ARM, where the tests are run via QEMU emulation.

2. Hermetic Windows Cross-Compilation

  • Problem: The previous build relied on implicit system paths to find Qt host tools (moc, rcc), causing failures when the runner environment didn't match the depends system exactly.
  • Fix: We now explicitly set QT_HOST_PATH and QT_HOST_PATH_CMAKE_DIR in the CMake configuration.
  • Result: The build now strictly uses the tools built in depends/<host>/native, completely ignoring the host OS environment. This guarantees consistency between CI and local builds.

3. Local Execution Support (act)

  • Problem: Running the full CI matrix locally (via act) spawned ~72 concurrent compilation threads, causing swap exhaustion and UI freezes on developer workstations.
  • Fix: Implemented dynamic concurrency control.
    • Added a Setup Concurrency step to all workflows.
    • Workflows now respect CMAKE_BUILD_PARALLEL_LEVEL if injected externally.
    • Added run-local-ci.sh wrapper that auto-scales threads (e.g., nproc / # of jobs) when running multiple jobs in parallel.

4. Test Stability (Wine/Windows)

  • Problem: Windows unit tests were crashing or hanging in the Linux CI environment.
  • Fix:
    • Enforced WINEARCH=win64 and installed wine64 packages to fix "Bad EXE format" errors.
    • Forced QT_QPA_PLATFORM=offscreen to prevent crashes when tests attempt to create GUI windows on headless runners.
    • Pre-initialized wineboot to prevent timeouts during test execution.

5. Packaging Fixes

  • Fix: Updated src/qt/CMakeLists.txt to use absolute paths (${CMAKE_CURRENT_SOURCE_DIR}) for icons and desktop files. This resolves CPack errors on macOS and Linux where relative paths were resolving incorrectly during the install phase.

5. Continuous Deployment

  • Feature: Added a deploy job that triggers only on version tags (v*). It aggregates artifacts from Linux/Windows/macOS builds and publishes a Draft Release with checksums.

Documentation:

  • New Guide: Added doc/build.md as the canonical build guide.
  • Migration: Added a detailed Autotools-to-CMake migration table
    explaining build types (RelWithDebInfo vs Release) and flags.
  • Platform Docs: - Updated doc/build-windows.md to feature WSL cross-compilation.
    • Deprecated doc/build-msys2.md and doc/build-unix.md.
    • Added experimental CMake instructions for macOS and BSDs.

The removal of autotools style build capability will be in in a separate PR.

@CyberTailor @div72 if you want to help out on this, please put up PR's against my source branch for this PR on my repo clone, and when I accept them they will flow to this one. In particular @div72, we need to adjust CI to add the windows builds to the cmake CI side and remove the depends builds from the non-cmake CI side.

Update: I have now done the core of the CI revisions. Please see the comments on the three new files below.

The complete removal of the non-cmake CI jobs should be reserved for the separate PR for the retirement of the autotools build for Gridcoin.

The old CI files have been deactivated and new CD has been integrated into CI.

Also, we need to get the Windows installer packaging working.

Integration of the existing share/setup.nsis.in script has been done with a custom cmake target "deploy".

@CyberTailor
Copy link
Contributor

Thanks! I have also made an attempt to make the depends system work with cmake (#2812 (comment)) but never finished it. Some results can be seen here: https://github.com/CyberTailor/Gridcoin-Research/tree/my-depends-cmake

Also, we need to get the Windows installer packaging working.

This seems to be easy: setup.nsi.in becomes NSIS.template.in and gets used by CPack. I've edited the template accordingly but haven't tested it yet.

When I have free time, I'll look at this.

@jamescowens
Copy link
Member Author

I have got Windows building correctly now and also checked native cmake compile for regression and it works. Going to work on Linux depends next. You and Div72 can handle the NSIS part and also the realignment of CI.

@jamescowens
Copy link
Member Author

jamescowens commented Nov 21, 2025

As of e31f7bb I have all three major targets working correctly with cmake.

Linux native build with cmake.
Linux depends build with cmake.
Windows depends build with cmake.

Also, here are the Gridcoin startup timings to show that the SSE/SHANI detection is working properly. Note the CMAKE_BUILD_TYPE flags must be used to get autotools default like performance, because cmake by default uses NO optimization, whereas autotools uses -02.

build_type_timings.xlsx

@jamescowens jamescowens force-pushed the depends_qt6 branch 9 times, most recently from a4c1737 to 22b7cfa Compare November 24, 2025 15:20
Updates dependency installation logic to support optional GUI builds
and better handle Qt versioning.

Changes to install_dependencies.sh:
- Added `WITH_GUI` parameter to `install_deps` function.
- Conditional Qt Installation:
  - Only installs Qt packages if `WITH_GUI="true"`.
  - Supports `USE_QT6="false"` on macOS by installing `qt@5`.
  - Skips Qt installation on Linux if `USE_QT6="false"`, printing a warning
    that Qt5 dependencies must be managed manually (due to EOL status on
    many distros).
- Ensures `PKGS_BASE` (build-essential, cmake, etc.) is always installed.

- Minor tweaks to other files to adjust.
jamescowens and others added 6 commits December 4, 2025 14:26
- Replace legacy build instructions with the modern CMake workflow.
- Add specific `pkg_add` dependency lists for both Headless and GUI builds.
- specific instructions for enabling Qt6 and XFCE integration.
- Include necessary system configuration steps (doas/sudo, wheel group, and messagebus).
Changes summary:
1. Add WINDRES
2. Make sure OBJCOPY and OBJDUMP are set
3. Make sure DEPENDS_COMPILE_DEFINITIONS are actually used
4. Add library search path to default LDFLAGS
5. Always restrict pkg-config search path to depends
6. Remove irrelevant conditions
@jamescowens
Copy link
Member Author

jamescowens commented Dec 6, 2025 via email

@jamescowens
Copy link
Member Author

jamescowens commented Dec 6, 2025

I want to make a point to the reviewers. Many of the tweaks over the past week have been due to testing in different scenarios to ensure that the builds work across a wide variety of conditions. This is not limited to the CI checks, which themselves have posed a challenge. This includes testing on bare iron development machines with many libraries installed (good hermeticity checks), and VM's of various operating systems.

The CI checks themselves have posed challenges in terms of the correct dependencies. Additional complexity was generated by ensuring that the CI works in both act and Github actions, because the base runner images in Github are slightly different than the base containers used in act.

@CyberTailor
Copy link
Contributor

If I want to add my commits to this PR, should I sumbit each PR to the source branch individually or simply push new commits on top of a single PR?

@jamescowens
Copy link
Member Author

I have only a little bit more work to do on this unless you see something glaring. I am trying to get Ubuntu 22.04 to do the depends build for Windows (this is not for CI, since we use 24.04 in the runners). 22.04 has mingw 8, which has an incomplete set of headers for DirectX 12, and so is proving to be a big challenge with the Qt compile.

I also am pushing today updated documentation for OpenBSD and FreeBSD since I got both working using the native compile.

I am not going to do any more rebases or force pushes. The commits so far are the commits. Therefore you are good to base your work on the source branch for this PR.

I hope you agree this gives a good boost for us. I am sure you can see improvements to be made though.

@jamescowens
Copy link
Member Author

I don't think you have push permission to my branch on my fork, so you would have to do a PR to my fork at the moment.

We could also merge this and then you could do a fresh branch off of development. Quite frankly this should be merged, as it is much better than what we have currently and we need it for the next testnet builds to lead up to the Natasha release.

Also add wine package to ensure tests run correctly.
This commit fixes a logic error in the package download fallback mechanism within
funcs.mk.

Previously, fallback attempts (e.g., to S3 or bitcoind-depends-sources) requested
the upstream filename/URL `$(3)`. This works when the upstream name matches the
local name, but fails for packages that rely on query strings or renaming
(e.g., `CMakeLists.txt?h=6.7.3`).

Mirrors are populated with the sanitized, locally-stored filenames `$(4)`. This change
forces fallback attempts to request `$(4)` instead of `$(3)`, ensuring the request matches
the storage convention on the mirrors and preventing 403/404 errors during primary download
failures.
Use CPack to generate NSIS installer
This adds an OS_REF environment variable in the cmake_production.yml
file and then uses that at the front of the depends cache key. This
is to ensure that prior cached depends using a differnt runner OS
are invalidated rather than inappropriately reused.
…s builds

This is the final cleanup for the Ubuntu 22.04 issue.
This updates build.md to make sure the path is specified for the
Windows installer command, and provides an Ubuntu 22.04 workaround
procedure for building the win64 in Ubuntu 22.04.
@jamescowens jamescowens removed the request for review from div72 December 8, 2025 13:31
@jamescowens jamescowens merged commit 34c19ef into gridcoin-community:development Dec 8, 2025
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Package Gridcoin-Research as Flatpak Add Static Linux Builds to CD [feature request] appimage format of wallet

3 participants