Better tagging directory structure

Author: mathbunnyru

.github/workflows/docker-build-test-upload.yml

@@ -78,7 +78,7 @@ jobs:
 
       - name: Write tags file 🏷
         run: >
-          python3 -m tagging.write_tags_file
+          python3 -m tagging.apps.write_tags_file
           --registry ${{ env.REGISTRY }}
           --owner ${{ env.OWNER }}
           --short-image-name ${{ inputs.image }}
@@ -94,7 +94,7 @@ jobs:
 
       - name: Write manifest and build history file 🏷
         run: >
-          python3 -m tagging.write_manifest
+          python3 -m tagging.apps.write_manifest
           --registry ${{ env.REGISTRY }}
           --owner ${{ env.OWNER }}
           --short-image-name ${{ inputs.image }}

.github/workflows/docker-merge-tags.yml

@@ -68,7 +68,7 @@ jobs:
       - name: Merge tags for the images 🔀
         if: env.PUSH_TO_REGISTRY == 'true'
         run: >
-          python3 -m tagging.merge_tags
+          python3 -m tagging.apps.merge_tags
           --short-image-name ${{ inputs.image }}
           --variant ${{ inputs.variant }}
           --tags-dir /tmp/jupyter/tags/

.github/workflows/docker-tag-push.yml

@@ -62,7 +62,7 @@ jobs:
           path: /tmp/jupyter/tags/
       - name: Apply tags to the loaded image 🏷
         run: >
-          python3 -m tagging.apply_tags
+          python3 -m tagging.apps.apply_tags
           --registry ${{ env.REGISTRY }}
           --owner ${{ env.OWNER }}
           --short-image-name ${{ inputs.image }}

.github/workflows/docker-wiki-update.yml

@@ -41,7 +41,7 @@ jobs:
 
       - name: Update wiki 🏷
         run: >
-          python3 -m tagging.update_wiki
+          python3 -m wiki.update_wiki
           --wiki-dir wiki/
           --hist-lines-dir /tmp/jupyter/hist_lines/
           --manifests-dir /tmp/jupyter/manifests/

.github/workflows/docker.yml

@@ -30,6 +30,7 @@ on:
       - "!tagging/README.md"
       - "tests/**"
       - "!tests/README.md"
+      - "wiki/**"
       - "requirements-dev.txt"
   push:
     branches:
@@ -50,6 +51,7 @@ on:
       - "!tagging/README.md"
       - "tests/**"
       - "!tests/README.md"
+      - "wiki/**"
       - "requirements-dev.txt"
   workflow_dispatch:
 

CHANGELOG.md

@@ -3,6 +3,12 @@
 This changelog only contains breaking and/or significant changes manually introduced to this repository (using Pull Requests).
 All image manifests can be found in [the wiki](https://github.com/jupyter/docker-stacks/wiki).
 
+## 2025-02-21
+
+Affected: all images.
+
+- **Non-breaking:**: Better tagging directory structure ([#2228](https://github.com/jupyter/docker-stacks/pull/2228)).
+
 ## 2025-02-18
 
 Affected: all images.

Makefile

@@ -78,20 +78,20 @@ linkcheck-docs: ## check broken links
 
 hook/%: VARIANT?=default
 hook/%: ## run post-build hooks for an image
-	python3 -m tagging.write_tags_file \
+	python3 -m tagging.apps.write_tags_file \
 	  --registry "$(REGISTRY)" \
 	  --owner "$(OWNER)" \
 	  --short-image-name "$(notdir $@)" \
 	  --variant "$(VARIANT)" \
 	  --tags-dir /tmp/jupyter/tags/
-	python3 -m tagging.write_manifest \
+	python3 -m tagging.apps.write_manifest \
 	  --registry "$(REGISTRY)" \
 	  --owner "$(OWNER)" \
 	  --short-image-name "$(notdir $@)" \
 	  --variant "$(VARIANT)" \
 	  --hist-lines-dir /tmp/jupyter/hist_lines/ \
 	  --manifests-dir /tmp/jupyter/manifests/
-	python3 -m tagging.apply_tags \
+	python3 -m tagging.apps.apply_tags \
 	  --registry "$(REGISTRY)" \
 	  --owner "$(OWNER)" \
 	  --short-image-name "$(notdir $@)" \

docs/index.rst

@@ -35,6 +35,7 @@ Table of Contents
    :caption: Maintainer Guide
 
    maintaining/new-images-and-packages-policy
+   maintaining/tagging
    maintaining/tasks
 
 .. toctree::

docs/maintaining/tagging.md

@@ -0,0 +1,94 @@
+# Tagging and manifest creation
+
+The main purpose of the source code in [the `tagging` folder](https://github.com/jupyter/docker-stacks/tree/main/tagging) is to properly write tag files and manifests for single-platform images,
+apply these tags, and merge single-platform images into one multi-arch image.
+
+## What is a tag and a manifest
+
+A tag is a label attached to a Docker image identifying specific attributes or versions.
+For example, an image `jupyter/base-notebook` with Python 3.10.5 will have a full image name `quay.io/jupyter/base-notebook:python-3.10.5`.
+These tags are pushed to our [Quay.io registry](https://quay.io/organization/jupyter).
+
+A manifest is a description of important image attributes written in Markdown format.
+For example, we dump all `conda` packages with their versions into the manifest.
+
+## Main principles
+
+- All images are organized in a hierarchical tree.
+  More info on [image relationships](../using/selecting.md#image-relationships).
+- Classes inherit from `TaggerInterface` and `ManifestInterface` to generate tags and manifest pieces by running commands in Docker containers.
+- Tags and manifests are reevaluated for each image in the hierarchy since values may change between parent and child images.
+- To tag an image and create its manifest, run `make hook/<somestack>` (e.g., `make hook/base-notebook`).
+
+## Utils
+
+### DockerRunner
+
+`DockerRunner` is a helper class to easily run a docker container and execute commands inside this container:
+
+```{literalinclude} tagging_examples/docker_runner.py
+:language: py
+```
+
+### GitHelper
+
+`GitHelper` methods are run in the current `git` repo and give the information about the last commit hash and commit message:
+
+```{literalinclude} tagging_examples/git_helper.py
+:language: py
+```
+
+The prefix of commit hash (namely, 12 letters) is used as an image tag to make it easy to inherit from a fixed version of a docker image.
+
+## Taggers and Manifests
+
+### Tagger
+
+`Tagger` is a class that can be run inside a docker container to calculate some tag for an image.
+
+All the taggers are inherited from `TaggerInterface`:
+
+```{literalinclude} ../../tagging/taggers/tagger_interface.py
+:language: py
+```
+
+So, the `tag_value(container)` method gets a docker container as an input and returns a tag.
+
+`SHATagger` example:
+
+```{literalinclude} ../../tagging/taggers/sha.py
+:language: py
+```
+
+- `taggers/` subdirectory contains all the taggers.
+- `apps/write_tags_file.py`, `apps/apply_tags.py`, and `apps/merge_tags.py` are Python executable used to write tags for an image, apply tags from a file, and create multi-arch images.
+
+### Manifest
+
+`ManifestHeader` is a build manifest header.
+It contains the following sections: `Build timestamp`, `Docker image size`, and `Git commit` info.
+
+All the other manifest classes are inherited from `ManifestInterface`:
+
+```{literalinclude} ../../tagging/manifests/manifest_interface.py
+:language: py
+```
+
+- The `markdown_piece(container)` method returns a piece of markdown file to be used as a part of the build manifest.
+
+`AptPackagesManifest` example:
+
+```{literalinclude} ../../tagging/manifests/apt_packages.py
+:language: py
+```
+
+- `quoted_output(container, cmd)` simply runs the command inside a container using `DockerRunner.run_simple_command` and wraps it to triple quotes to create a valid markdown piece.
+  It also adds the command which was run to the markdown piece.
+- `manifests/` subdirectory contains all the manifests.
+- `apps/write_manifest.py` is a Python executable to create the build manifest and history line for an image.
+
+### Images Hierarchy
+
+All images' dependencies on each other and what taggers and manifest are applicable to them are defined in `hierarchy/images_hierarchy.py`.
+
+`hierarchy/get_taggers_and_manifests.py` defines a function to get the taggers and manifests for a specific image.

docs/maintaining/tagging_examples/docker_runner.py

@@ -0,0 +1,4 @@
+from tagging.utils.docker_runner import DockerRunner
+
+with DockerRunner("ubuntu") as container:
+    DockerRunner.run_simple_command(container, cmd="env", print_result=True)