Cached native runners / build scripts for all target platforms #106
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Phatcat
force-pushed
the
cached_runners
branch
2 times, most recently
from
96ec6ef to
829b237
Compare
|
Tested the ubuntu builder and... 35 seconds vcpkg cache pre-loading 06 min 40 seconds: booting/preloading before reaching the vcpkg call in build_linux.sh cache not saved because no changes... would take 30 - 60 seconds depending on size. So the full vcpkg solution takes about 2 minutes with loading and storing, with only loading and no storing, it would be closer to 1 minute... So native linux build will be 15+ minutes regardless... Okay, so windows will keep the vcpkg approach until a native ubuntu builder becomes faster than 13 minutes (hopefully with 25) |
Phatcat
changed the title
|
Status update on native linux build, I was wrong, it actually spent much more than 6 minutes alone on vcpkg, unlike windows that only spends a few minutes... Using homebrew on linux x64 (arm64 coming soon as well) proves to be a much faster way of handling dependencies that aren't available through apt-get, especially boost and botan. Down to about 11 min building now. |
Phatcat
force-pushed
the
cached_runners
branch
2 times, most recently
from
2d61b3e to
1780e44
Compare
|
6 min vcpkg native windows buildrunner incoming.
|
Phatcat
force-pushed
the
cached_runners
branch
4 times, most recently
from
48e79ce to
0d819fa
Compare
Phatcat
changed the title
Complete Botan-3 migration target: Removal of FindBotan and using supplied config Migrate from marked as deprecated functions -1363 padding needs to be done manually now -No serialization directly on the hasher -no declaration of functions with a serialization. target_link_libraries: srp6 need botan and doesn't have it linked libs/shared need to be linked to botan as well
Patch the botan config file for libbotan-3-dev. It is expecting the header files in a wrong location Same goes for the library files, it is not considering multi architecture.
Get rid of FindMySQL and use provided config-file. Update code to reflect correct include structure libmysqlcppconn-dev is a hacky uncomplete unbuntu library with custom file staging that's incompatible with modern setups (include/jdbc) so grab official binaries instead and install directly into cmake. Require at least vlersion 8.3 as it's the first version to come with the config for the proper include structure.
Complete rewrite of FindPCRE.cmake module from scratch; - Properly allocate both release and debug libraries - Automatically prefer static libraries if found - Properly check the set required version and use find_package_handle_standard_Args to throw an error if the version isn't matching or higher than requested - Report which library (or several) are in use and if they're static or shared
Implement missing handling of toUpper on char32_t on libcpp+
Getting this error on ubuntu-arm buildrunner: ``` 70.52 /usr/src/ember/src/libs/shared/shared/threading/Spinlock.h: In member function 'void ember::Spinlock::lock()': 70.52 /usr/src/ember/src/libs/shared/shared/threading/Spinlock.h:20:27: error: '__yield' was not declared in this scope 70.52 20 | #define YIELD_INSTRUCTION __yield() 70.52 | ^~~~~~~ 70.52 /usr/src/ember/src/libs/shared/shared/threading/Spinlock.h:54:33: note: in expansion of macro 'YIELD_INSTRUCTION' 70.52 54 | YIELD_INSTRUCTION; 70.52 | ```
Windows is building, no one touch it
linux thorugh vcpkg is building, no one touch it..
Hitting all 3 target platforms with debug builds targeting both x64 and arm64, just like the upcoming manual build guide. - The windows and linux build tests the vcpkg configuration and reveals any issues in that install path. - macOS uses a homebrew approach and only needs to get the binaries from mysql qhich takes about a second. Add a simple check in the top level cmakefile and add homebrew libraries if we're on apple so the buildbot and people can use natively installed libraries. - The linux docker already tests a native dpendency path, and most packages can be got from apt-get with the exception of a proper mysql, but that takes but a second to download and install. all caches are about 1.1gb in size, and with the current 4 native builders that's about 4gb in total so there's room for expansion
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on top of (requires) #86 #90
#94#95 #105#108Addressing #72 #73 #74
- MPQ extraction
- vcpkg (mysql)
- vcpkg (mysql)
The changes getting PR'd here are the 3 commits
We got clang compiling, msvc compiling, gcc compiling; windows (not arm yet though), macOS (only apple silicon) and ubuntux64 and arm. vcpkg is a valid path for all (except for windows arm atm)
vcpkg cache creation takes anywhere between 30 and 100 minutes, but once it's set up it's self updating on every build and takes maybe 1 minute to load up and half a minute to store. Currently the only reliable (for x64) way to install the dependencies for windows completely from CLI.
The failing mysql package from vcpkg has been reported: microsoft/vcpkg#45378
Things to improve:
Currently macOS takes 5 minutes, dockerfile takes 6 minutes, but native windows with vcpkg takes 13 minutes while native ubuntu with vcpkg takes 17 minutes... 17 minutes is a long time to wait compared to 5/6 minutes...
Perhaps macOS needs to be the vcpkg target if it's the faster of the 3, it's just a shame to take the fastest buildbot and make it the slowest..
Would also only make a difference if it's possible to even get a native ubuntu runner down to less than 13 minutes for a full build and test run. It seems vastly slower running natively than when running a docker-image, but the caching of docker is not natively supported by github and using ghcr seems to be a mess with organization repos..