add support for generating Singularity container recipes & (optionally) images via 'sudo singularity' (experimental feature) #2332
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ages (WIP, initial structure) (#1)
…cal Modules and Flat Naming Scheme. Definition files are stored in $EASYBUILD_PACKAGEPATH. Need thorough testing for other module naming schemes.
Contributor
Author
The generated definition file |
Member
|
@shahzebsiddiqui This seems to be working on top of my commit in #2263, but somehow my commit is not included in your branch? That's a bit confusing... Maybe you should issue a PR to my branch I used for #2263 instead ( In any case, the approach you took is way too hardcoded, we need to make it more flexible so people have some control over which base image is used, etc. Also, I'd like to make it a self-contained What we probably need is support to let people provide a template I'm also having second thoughts on leveraging In that sense, this is not a package at all, it's a recipe for a different tool that happens to use EasyBuild in it. So, we probably need a dedicated option for this, rather than leveraging If we're still interested in letting EasyBuild call out to |
shahzebsiddiqui
force-pushed
the
package_singularity
branch
from
41854f1 to
75cfae2
Compare
alter bootstrap option when generating def file. Easybuild will write definition file to $EASYBUILD_PACKAGEPATH for now. After definition file is created it will try to build the container using sudo singularity build <image> def_file. This process assumes the user has sudo priviledge to run this operation
shahzebsiddiqui
force-pushed
the
package_singularity
branch
from
75cfae2 to
aabdefa
Compare
…uild this container in shub and made easybuild reference my container. More work needs to be done with tagging containers in shub to allow multiple container types and option for building from different bootstrap process. Also work needs to be done to figure out os dependency and automatically add them in recipe and install the packages before installing app.
Contributor
Author
|
@boegel Singularity def and image file Testing bzip2 |
Contributor
Author
|
For your reference, I have set shub as the default bootstrap and centos as the primary os. You may note that OS release is not specified that maybe an option that user can pass. Right now I am using centos image from docker official channel since there is not an official centos image in shub. |
…evel package. Rename easybuild bootstrap file. SHUB uses extension for tagging so centos image will have tag centos-7.3.1611
shahzebsiddiqui
force-pushed
the
package_singularity
branch
from
3b099ba to
4b70766
Compare
…ase to control container properties. By default eb will generate recipe file, but to enable container build from eb you must pass --build-container. Default container size is 1024, eb will do singularity image.create but in reality container will be resized in 2.4. --container-size will be useful for singularity <2.4 where you must use singularity create first before bootstraping container. %labels section adds architecture name, host and cpu model. Architecture name is calculated by finding the model number.
Contributor
Author
|
I have created a new container in shub with tag "centos-7.3.1611". See https://www.singularity-hub.org/containers/567 Shown below is the latest changes with %labels that queries system info that can be queried by singularity New flags in eb for singularity In order to use different OS and release we would need to create another bootstrap easybuild recipe and build in shub. This can be extended for Docker. Next step would be to get os-dependency installed in %post section and then start installing toolchain containers in shub. We can bootstrap from shub if we provide a set of containers with different tags GCCcore-5.4.0, foss-2016b, gompi-2016b, ... eb should be able to figure out the the Bootstrap line when generating recipe when --toolchain is specified. |
Contributor
Author
|
@boegel Adding support for localimage bootstrap |
…alimage bootstrap option. Image must be of extension .img or .simg according to singularity. Its up to user to ensure the local image is built correctly when using it as a bootstrap agent
shahzebsiddiqui
force-pushed
the
package_singularity
branch
from
5542764 to
c5ae26c
Compare
…l build this in shub
shahzebsiddiqui
force-pushed
the
package_singularity
branch
from
23e6990 to
aacd372
Compare
creating custom image name during singularity build.
…rectories in /scratch in container to strip everything out. /scratch still needs to be present in container for bootstraping again since --prefix requires /scratch to be created in advance. Toolchain container boostrap progression: GCCcore-5.4.0 -> GCC-5.4.0-2.26 -> gompi-2016b -> foss-2016b
Contributor
Author
|
Support for --image-name which can override default naming scheme for images. |
to ensure this builds properly with Travis
Add option --container-path for controlling where to write container images.
container formats (squashfs, ext3, sandbox). This option only affects when building container using --build-container=True. .simg is used with squashfs, while .img is used with ext3
Contributor
Author
|
Singularity Options |
Contributor
Author
|
Using example --container-path |
Contributor
Author
|
example with ext3 image format |
reworked implementation of Singularity integration
houndci-bot
reviewed
Apr 19, 2018
| import test.framework.asyncprocess as a | ||
| import test.framework.build_log as bl | ||
| import test.framework.config as c | ||
| import test.framework.containers as ct |
There was a problem hiding this comment.
module level import not at top of file
| """ | ||
| return ConfigurationVariables()['packagepath'] | ||
|
|
||
| def container_path(): |
| install_os_deps = '' | ||
| for ec in easyconfigs: | ||
| for osdep in ec['ec']['osdependencies']: | ||
| if isinstance(osdep, basestring): |
houndci-bot
reviewed
Apr 19, 2018
| import test.framework.asyncprocess as a | ||
| import test.framework.build_log as bl | ||
| import test.framework.config as c | ||
| import test.framework.containers as ct |
There was a problem hiding this comment.
module level import not at top of file
| """ | ||
| return ConfigurationVariables()['packagepath'] | ||
|
|
||
| def container_path(): |
| install_os_deps = '' | ||
| for ec in easyconfigs: | ||
| for osdep in ec['ec']['osdependencies']: | ||
| if isinstance(osdep, basestring): |
shahzebsiddiqui
changed the title
style fixes to make Hound CI happy
houndci-bot
reviewed
Apr 19, 2018
test/framework/suite.py
Outdated
| import test.framework.variables as v | ||
| import test.framework.yeb as y | ||
|
|
||
| # set plain text key ring to be used, so a GitHub token stored in it can be obtained without having to provide a password |
There was a problem hiding this comment.
line too long (121 > 120 characters)
houndci-bot
reviewed
Apr 19, 2018
test/framework/suite.py
Outdated
| import test.framework.variables as v | ||
| import test.framework.yeb as y | ||
|
|
||
| # set plain text key ring to be used, so a GitHub token stored in it can be obtained without having to provide a password |
There was a problem hiding this comment.
line too long (121 > 120 characters)
fix some more style issues in test/framework/suite.py
boegel
changed the title
boegel
approved these changes
Apr 19, 2018
Member
|
Thanks for all your work on this @shahzebsiddiqui, and @FooBarQuaxx for the valuable comments! This initial support for Singularity integration will be a part of the next EasyBuild release (3.6.0), which should become available in the next couple of days... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
edit (by @boegel): see new command line/configuration options:
--containerize/-Cto enable generating of container recipes in path specified by--containerpath--container-*options to control the behavior--container-build-imageto enable building of container image with generating recipe by calling out tosudo singularity--container-image-formatto specify desired image of container images to be built (ext3,sandboxorsquashfs(default))--container-image-nameto control name of generated container recipe & imageCurrently relies on a base container image being used (to be specified via
--container-base), see https://singularity-hub.org/collections/143.Requires use of
--experimentalsince the functionality is subject to change.Documentation is in the works via easybuilders/easybuild#414