Automate release process

devel/README.md

@@ -21,9 +21,14 @@ development project.
 ### Used tools
 
 The script requires the `git`, `gh`, `jq` and `osc` command line tools to be installed. The `osc`
-and `gh` tools need to be configured/authenticated against OBS respective GitHub. Do not worry the
+and `gh` tools need to be configured/authenticated against OBS and GitHub, respectively. Do not worry the
 script checks for that.
 
+Use the following commands if you don't want to remember your usernames:
+
+    GH_USER=$(gh auth status | sed -n 's/.*Logged in to github.com account \([^[:space:]]*\) .*/\1/;T;p;q')
+    OBS_USER=$(osc user | sed "s/^\([^:]*\):.*$/\\1/")
+
 ### GitHub configuration
 
 If you run the script in your GitHub fork then you need to configure the OBS credentials. When
@@ -33,29 +38,29 @@ to change anything.
 You need to create the `OBS_USER` action variable containing your OBS login name. You can do that
 from command line running command
 
-    gh -R <gh_user>/agama variable set OBS_USER --body <obs_user>
+    gh -R ${GH_USER?}/agama variable set OBS_USER --body ${OBS_USER?}
 
-where `gh_user` is your GitHub login name and `obs_user` your OBS login name.
+where `$GH_USER` is your GitHub login name and `$OBS_USER` your OBS login name.
 
 Alternatively you can create the variable manually by visiting URL
 
-    https://github.com/<gh_user>/agama/settings/variables/actions/new
+    xdg-open https://github.com/${GH_USER?}/agama/settings/variables/actions/new
 
-where `gh_user` is your GitHub login name. On that page create variable `OBS_USER` with your OBS
+where `$GH_USER` is your GitHub login name. On that page create variable `OBS_USER` with your OBS
 login name.
 
 Similarly we need to enter the OBS password, but as this is sensitive private value we use GitHub
 secret for that. From command line run this command
 
-    gh -R <gh_user>/agama secret set OBS_PASSWORD
+    gh -R ${GH_USER?}/agama secret set OBS_PASSWORD
 
-where `gh_user` is your GitHub login name. The command will interactively ask for the password.
+where `$GH_USER` is your GitHub login name. The command will interactively ask for the password.
 
 Or you can create the secret in browser going to this page
 
-    https://github.com/<gh_user>/agama/settings/secrets/actions/new`
+    xdg-open https://github.com/${GH_USER?}/agama/settings/secrets/actions/new`
 
-where `gh_user` is your GitHub login name. Create a new secret with name `OBS_PASSWORD` and
+where `$GH_USER` is your GitHub login name. Create a new secret with name `OBS_PASSWORD` and
 enter your OBS password as the value.
 
 ### Testing builds
@@ -83,7 +88,7 @@ project for submitting.
   - `git checkout master`
   - Update the ISO version in `live/src/agama-installer.kiwi`, use the `pre` suffix to distinguish
     between a development version and the final version. I.e. for Beta3 change the version from `12`
-    to `13pre`.
+    to `13.pre`.
   - Push the changes
     - `git commit -a`
     - `git push`
@@ -126,3 +131,33 @@ GitHub variable. It is in JSON format and maps the Git branch name to the OBS pr
 
 The GitHub submission actions check the mapping value for the current branch/tag and if no mapping
 is found the submission is skipped.
+
+## Detailed implementation
+
+- check if needed tools are installed, else terminate (git gh jq osc)
+- check if user is authenticated to gh and osc, else terminate
+
+- modes of operation:
+  - print obsolete projects (those where the branch no longer exists)
+  - cleanup obsolete projects
+  - setup a project for submission (described in the following)
+
+- if setting up a fork, check that OBS_USER and OBS_PASSWORDS are defined,
+  else terminate
+
+- if the OBS project does not exist yet, create it:
+  - branch a **set of packages** from systemsmanagement:Agama:Devel to the target project
+  - disable building for archs: aarch64, i586, ppc64le and s390x (unless `-a` is given)
+  - disable building for targets: Leap 16.x (unless `-t` is given)
+  - for package agama-installer-Leap, disable building for target: images(TW)
+  - enable publishing the packages and images (which is off by default in OBS)
+  - set the project url, title, description: to point to the git repo and branch
+
+- update gh variable OBS_PROJECTS for the repo to include this branch->project
+
+- for each of the set of **CI Workflows** (last major change: PR#2145)
+  - trigger the workflow
+    - if the remote branch already exists, do it
+    - otherwise print the command to do it
+
+- print the URL of the OBS project

devel/branch2obs.sh

@@ -1,20 +1,27 @@
 #! /usr/bin/bash
 
-# This script configures autosubmission from a GitHub branch to an OBS project.
-# Works with the original project and with forks as well.
-
 usage () {
+  echo "This script configures autosubmission from a GitHub branch to an OBS project."
+  echo "Works with the original project and with forks as well."
+  echo
   echo "Usage: $0 [options]"
   echo
   echo "Options:"
   echo "  -a              - keep all original build archs (default: build only x86_64)"
   echo "  -b <branch|tag> - source git branch or tag (default: current git branch)"
-  echo "  -p <project>    - target OBS project (based on the git branch)"
-  echo "  -t              - keep all original build targets (default: disable Leap 16.0)"
-  echo "  -c              - cleanup (delete) all obsolete projects, exclusive option,"
-  echo "                    all other options are ignored"
+  echo "  -p <project>    - target OBS project"
+  echo "                    (default for main repo:"
+  echo "                               systemsmanagement:Agama:branches:\$BRANCH or"
+  echo "                               systemsmanagement:Agama:Devel  for master"
+  echo "                             for forks:"
+  echo "                               home:\$osc_user:Agama:branches:\$BRANCH)"
+  echo "  -t              - keep all original build targets (default: disable Leap 16.x)"
+  echo
+  echo "  -c              - cleanup (delete) all obsolete projects (branch no longer exists);"
+  echo "                    exclusive option, all other options are ignored"
   echo "  -o              - print obsolete projects, similar to -c but only print"
   echo "                    the projects instead of deleting them"
+  echo "                    Obsolete are those that don't have a git branch anymore."
   echo "  -h              - print this help"
 }
 
@@ -81,6 +88,7 @@ if ! gh auth status --active > /dev/null 2>&1; then
   exit 1
 fi
 
+# "agama-project/agama" or similar for a fork
 repo_slug=$(gh repo view --json nameWithOwner -q ".nameWithOwner")
 
 if [ -n "$CLEANUP" ] || [ -n "$OBSOLETE" ]; then
@@ -103,7 +111,7 @@ if [ -n "$CLEANUP" ] || [ -n "$OBSOLETE" ]; then
   for project in "${obs_projects[@]}"; do
     # remove the prefix to get the related Git branch
     branch_name=$(echo -n "$project" | sed -e "s/^$prefix://")
-    if ! git ls-remote --exit-code --heads origin "$branch_name" > /dev/null; then
+    if ! git ls-remote --exit-code --branches origin "$branch_name" > /dev/null; then
       if [ -n "$CLEANUP" ]; then
         echo "Deleting obsolete project $project..."
         # recursive remote delete (allows deleting a non-empty project)
@@ -141,6 +149,10 @@ if [ "$repo_slug" = "agama-project/agama" ]; then
       PROJECT="systemsmanagement:Agama:branches:${BRANCH}"
     fi
   fi
+  # BTW, var: OBS_USER and secret: OBS_PASSWORD for the main repo
+  # are defined at the organization level, not repo level. See:
+  #     gh variable list -o agama-project
+  #     gh secret   list -o agama-project
 else
   echo "GitHub fork detected"
 
@@ -174,7 +186,14 @@ if osc ls "$PROJECT" > /dev/null 2>&1; then
 else
   echo "Creating project $PROJECT..."
   # packages to branch
-  packages=(agama agama-installer agama-auto agama-products agama-web-ui rubygem-agama-yast)
+  packages=(
+    agama
+    agama-installer
+    agama-auto
+    agama-products
+    agama-web-ui
+    rubygem-agama-yast
+    )
   for pkg in "${packages[@]}"; do
     echo "Branching package $pkg"
     # branch the package
@@ -189,34 +208,45 @@ else
     echo "Disabling build on aarch64, i586, ppc64le and s390x"
     osc meta prj "$PROJECT" | \
       sed "/<arch>aarch64<\/arch>/d;/<arch>i586<\/arch>/d;/<arch>ppc64le<\/arch>/d;/<arch>s390x<\/arch>/d;" | \
-      osc meta prj -F - "$PROJECT"
+      osc meta prj --file - "$PROJECT"
   fi
 
-  # disable Leap 16.0 target
+  # disable Leap 16.x targets
   if [ "$ALL_TARGETS" != true ]; then
-    echo "Disabling openSUSE Leap 16.0 build target"
+    echo "Disabling openSUSE Leap 16.x build targets"
+    ADD='<build>
+        <disable repository="openSUSE_Leap_16.1"/>
+        <disable repository="openSUSE_Leap_16.0"/>
+        <disable repository="images_Leap_16.1"/>
+        <disable repository="images_Leap_16.0"/>
+      </build>'
+    # remove newlines to fit into a sed command
+    ADD="$(echo "$ADD" | tr '\n' ' ')"
     osc meta prj "$PROJECT" | \
-      sed 's#</description>#</description><build><disable repository="openSUSE_Leap_16.0"/><disable repository="images_Leap_16.0"/></build>##' | \
-      osc meta prj -F - "$PROJECT"
+      sed "s#</description>#</description>$ADD#" | \
+      osc meta prj --file - "$PROJECT"
   fi
 
   # disable building the agama-installer-Leap image for Tumbleweed, that does not work
+  echo "Disabling build of agama-installer-Leap for TW"
+  ADD='<build>  <disable repository="images"/>  </build>'
   osc meta pkg "$PROJECT" agama-installer-Leap | \
-    sed 's#</package>#<build><disable repository="images"/></build></package>##' | \
-    osc meta pkg -F - "$PROJECT" agama-installer-Leap
+    sed "s#</package>#$ADD</package>#" | \
+    osc meta pkg --file - "$PROJECT" agama-installer-Leap
 
   # enable publishing of the built packages and images (delete the disabled publish section)
   echo "Enable publishing of the build results"
   osc meta prj "$PROJECT" | sed "/^\s*<publish>\s*$/,/^\s*<\/publish>\s*$/d" | \
-    osc meta prj -F - "$PROJECT"
+    osc meta prj --file - "$PROJECT"
 
   echo "Set project description"
   url=$(gh repo view --json url --jq .url)
+  DESCR="This project contains the latest packages built from repository $repo_slug, branch \"$BRANCH\"."
   osc meta prj "$PROJECT" | 
     sed -e "s#<url>.*</url>#<url>$url/tree/$BRANCH</url>#" \
       -e "s#<title>.*</title>#<title>Agama from Git</title>#" \
-      -e "s#<description>.*</description>#<description>This project contains the latest packages built from repository $repo_slug, branch \"$BRANCH\".</description>#" | \
-    osc meta prj -F - "$PROJECT"
+      -e "s#<description>.*</description>#<description>$DESCR</description>#" | \
+    osc meta prj --file - "$PROJECT"
 fi
 
 # configure OBS_PROJECTS GitHub variable
@@ -228,12 +258,20 @@ if [ -z "$projects" ]; then
 fi
 
 # insert the mapping for the new branch
-echo "$projects" | jq ". += { \"$BRANCH\" : \"$PROJECT\" } " | gh -R "$repo_slug" variable set OBS_PROJECTS
+echo "$projects" |
+  jq --sort-keys ". += { \"$BRANCH\" : \"$PROJECT\" } " |
+  gh -R "$repo_slug" variable set OBS_PROJECTS
 
 # to really synchronize the GitHub content with OBS trigger the autosubmission jobs if the remote
 # brach already exists or print the instructions for later
-workflows=(obs-staging-live.yml obs-staging-products.yml obs-staging-rust.yml obs-staging-service.yml obs-staging-web.yml)
-if git ls-remote --exit-code --heads origin "$BRANCH" > /dev/null; then
+workflows=(
+  obs-staging-live.yml
+  obs-staging-products.yml
+  obs-staging-rust.yml
+  obs-staging-service.yml
+  obs-staging-web.yml
+  )
+if git ls-remote --exit-code --branches origin "$BRANCH" > /dev/null; then
   for workflow in "${workflows[@]}"; do
     echo "Starting GitHub Action $workflow..."
     gh workflow run "$workflow" --ref "$BRANCH"
@@ -249,3 +287,4 @@ fi
 
 echo
 echo "Git branch \"$BRANCH\" is now automatically submitted to OBS project \"$PROJECT\""
+echo "  xdg-open https://build.opensuse.org/project/show/$PROJECT"