Cannot release package with kctrl

[mamachanko]

What steps did you take:

using registry/project/image as a placeholder for my actual image

I have a directory containing the to-be contents of my package:

config
  |_ upstream # this is an imgpkg bundle, i.e. has upstream/.imgpkg
  |_ downstream # this is plain YTT 

I ran

kctrl pkg init

Then I adjusted PackageBuild manually:

apiVersion: kctrl.carvel.dev/v1alpha1
kind: PackageBuild
metadata:
  name: my.package.com
spec:
  release:
  - resource: {}
  template:
    spec:
      export:
      - imgpkgBundle:
          image: registry/project/image
          useKbldImagesLock: false
        includePaths:
        - config/downstream
        - config/upstream
      app:
        spec:
          deploy:
          - {}
          template:
          - ytt:
              paths:
              - .
          - kbld: {}

Then I updated the resources in package-resources.yaml to reflect my needs:

• Filled in metadata
• Set Package

apiVersion: data.packaging.carvel.dev/v1alpha1
kind: Package
metadata:
 name: my.package.com.0.0.0
spec:
 refName: my.package.com
 template:
   spec:
     deploy:
       - kapp:
           rawOptions:
             - --wait-timeout=5m
             - --kube-api-qps=20
             - --kube-api-burst=30
     fetch:
       - git: { }
     template:
       - ytt:
           paths:
             - config/upstream
             - config/downstream
       - kbld: { }
 valuesSchema:
   openAPIv3: null
 version: 0.0.0

What happened:
When I run

kctrl pkg release -v 0.0.0-$USER

it fails with

❯ kctrl pkg release -v 0.0.0-$USER --chdir manifests/build

Prerequisites
1. The host must be authorized to push images to a registry (can be set up by
running `docker login`)
2. Package can be released only after `kctrl package init` has been run
successfully.

The bundle created needs to be pushed to an OCI registry. Registry URL format:
<REGISTRY_URL/REPOSITORY_NAME> e.g. index.docker.io/k8slt/sample-bundle
> Enter the registry URL (regisrty/project/path):
W1019 09:12:46.033515   65993 gcp.go:119] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.26+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke

Target cluster 'https://35.195.245.51' (nodes: gke-tinwork-whinny-default-pool-664656e5-3xmv, 4+)

kbld builds images when necessary and ensures that all image references are
resolved to an immutable reference
Building images and resolving references
            | $ ytt -f /var/folders/0r/_173x1jn2d1bp7h4twh90hc80000gp/T/kapp-controller-fetch-template-deploy1953851271/0
            | $ kbld -f - --imgpkg-lock-output=.imgpkg/images.yml

kctrl: Error: Reconciling: kbld:
  Error:
- Resolving image 'regisrty/project/image': GET https://regisrty/v2/project/image/manifests/latest: NOT_FOUND: repository project/image not found

Annoyingly, my package-build.yaml gets overwritten by kctrl every time I run the release command:

• It reformats it
• It reorders fields
• It adds creationTimestamp: null
• It removes useKbldImagesLock: false

If there are other (unrelated) YTT files in my repository, it fails as it tries to template them.

What did you expect:

That's hard to say. I think I would expect my package image to be pushed and Package resource to appear somewhere.

Anything else you would like to add:

I am switching back to scripted package releasing.

Environment:

❯ kctrl version
kctrl version 0.41.2

Succeeded

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[100mik]

This is definitely not expected. We try to maintain user supplied values, gonna take a closer look a it.

[100mik]

Annoyingly, my package-build.yaml gets overwritten by kctrl every time I run the release command:

This seems to be true, and might be a bit buggy.
This seems to be happening the command seems to reduce the config to the most minimal version (not intended at all)
Even if, useKbldImagesLock is removed it defaults to false, so it should not affect functionality of the release command itself.
This might be improved for a better user experience, we could not overwrite the file if there are no changes, or improve how we handle marshalling/unmarshalling.

I did however find some other buggy behaviour, the .imgpkg folder seems to be deleted after a run with useKbldImagesLock set to false. This definitely should not be the case.

[100mik]

The output you shared however seems to be from when useKbldImagesLock output is set to true as it is trying to build images, which confuses me a bit.
Was this run after you set the value to false?

More so as of today,
kctrl expects the user supplied .imgpkg directory to reside in the root, so something like

.
|-.imgpkg
|-upstream
|-downstream

Thought I would call that out 🤔

[joostvdg]

Using the latest version I could find (0.44.1) this still happens:

❯ kctrl version

kctrl version 0.44.1

Succeeded

Created the initial package via kctrl package init.

If I set useKbldImagesLock to true, it works as expected (image lock file generated, package released).
Setting it to false gives unexpected behavior:

1. the release fails, This directory is not a bundle. It is missing .imgpkg/images.yml
2. the useKbldImagesLock: false line is removed from the package-build.yml

ps. the reason I want to disable the lock file, is that I have a multi-arch image, and the lock file is only for a single arch, which defeats the purpose of multi-arch 🤷

[joostvdg]

The add more information to my previous comment.

Here's the package-build.yml:

apiVersion: kctrl.carvel.dev/v1alpha1
kind: PackageBuild
metadata:
  creationTimestamp: null
  name: client.gitstafette.kearos.net
spec:
  release:
  - resource: {}
  template:
    spec:
      app:
        spec:
          deploy:
          - kapp: {}
          template:
          - ytt:
              paths:
              - config
          - kbld: {}
      export:
      - imgpkgBundle:
          image: ghcr.io/joostvdg/gitstafette/client-bundle
          useKbldImagesLock: false
        includePaths:
        - config

I can work around this by generating an Image Lock File via kbld before running kctrl package release, though this kinda sucks.
But, for that to work, I have to 1) fix my bundle to a specific platform (not desired), 2) add additional config to the package-build.yml, and 3) re-create the .imgpkg folder before running kbld.

      export:
      - imgpkgBundle:
          image: ghcr.io/joostvdg/gitstafette/client-bundle
        includePaths:
        - config
        - .imgpkg/images.yml

My kbld command is as follows:

mkdir -p packages/gitstafette-client/.imgpkg
	ytt -f packages/gitstafette-client/config | kbld -f - --platform linux/arm64 --imgpkg-lock-output packages/gitstafette-client/.imgpkg/images.yml

In both scenarios, useKbldImagesLock: false is removed, and the folder .imgpkg is deleted.

[100mik]

Related #963

[100mik]

@joostvdg you do need a lock file while releasing a Package bundle.

The intention of the useKbldImagesLock is for users who want to supply their own lock file. I am not very familiar with the guidance about lock files for multi-arch images or if it is possible (cc: @joaopapereira ).

Is the intention here to supply an empty images lock file so that no images are locked on to?

[100mik]

Ran into a tricky spot with this one, we are still running kbld and overwriting the user supplied images lock no matter what. Something we definitely do not want. This does mean that the reconciler (which runs ytt and then kbld initially) needs to be aware of whether the export section has useKbldImagesLock.

Makes this a bit tricky, giving it some thought. Even though the current changes achieve the goal of this PR. The useKbldImagesLock option is not exactly functional. Should we include that within the goals of this PR? (cc: @praveenrewar )