Creation timestamp set to invalid value "null" when using kustomize v5.0.0

[abhay-krishna]

What happened?

I was trying to generate manifests for a cluster-api-provider project (tinkerbell/cluster-api-provider) and installed kustomize using the install_kustomize script. But the generated manifests had some spurious values for creationTimestamp

➜ kustomize version
v5.0.0
➜ kustomize build config/default | grep "creationTimestamp"
  creationTimestamp: "null"
  creationTimestamp: "null"
  creationTimestamp: "null"
  creationTimestamp: null
  creationTimestamp: null
  creationTimestamp: null

Note the quotes around null. This is causing issues when trying to parse time in a client-go applicatio.

Error: unable to convert unstructured object to apiextensions.k8s.io/v1, Kind=CustomResourceDefinition: parsing time "null" as "2006-01-02T15:04:05Z07:00": cannot parse "null" as "2006"

However I don't see this issue in an older version of kustomize.

➜ kustomize version
{Version:kustomize/v4.5.5 GitCommit:daa3e5e2c2d3a4b8c94021a7384bfb06734bcd26 BuildDate:2022-05-20T20:25:40Z GoOs:darwin GoArch:amd64}
➜ kustomize build config/default | grep "creationTimestamp"
  creationTimestamp: null
  creationTimestamp: null
  creationTimestamp: null

What did you expect to happen?

I expect that the generated manifest should conform to types of metav1.ObjectMeta where creationTimestamp is a Time object and not string, i.e., null instead of "null".

How can we reproduce it (as minimally and precisely as possible)?

kustomization.yaml under config/crd in https://github.com/tinkerbell/cluster-api-provider-tinkerbell. Commenting out some of the CRDs to reproduce minimally

# This kustomization.yaml is not intended to be run by itself,
# since it depends on service name and namespace that are out of this kustomize package.
# It should be run by config/default
commonLabels:
  cluster.x-k8s.io/v1beta1: v1beta1
resources:
- bases/infrastructure.cluster.x-k8s.io_tinkerbellclusters.yaml
#- bases/infrastructure.cluster.x-k8s.io_tinkerbellmachines.yaml  # THIS LINE HAS BEEN COMMENTED OUT FOR SAKE OF MINIMALITY
#- bases/infrastructure.cluster.x-k8s.io_tinkerbellmachinetemplates.yaml  # THIS LINE HAS BEEN COMMENTED OUT FOR SAKE OF MINIMALITY
# +kubebuilder:scaffold:crdkustomizeresource

patchesStrategicMerge:
# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix.
# patches here are for enabling the conversion webhook for each CRD
- patches/webhook_in_tinkerbellclusters.yaml
#- patches/webhook_in_tinkerbellmachines.yaml
#- patches/webhook_in_tinkerbellmachinetemplates.yaml
# +kubebuilder:scaffold:crdkustomizewebhookpatch

# [CERTMANAGER] To enable webhook, uncomment all the sections with [CERTMANAGER] prefix.
# patches here are for enabling the CA injection for each CRD
- patches/cainjection_in_tinkerbellclusters.yaml
#- patches/cainjection_in_tinkerbellmachines.yaml
#- patches/cainjection_in_tinkerbellmachinetemplates.yaml
# +kubebuilder:scaffold:crdkustomizecainjectionpatch

# the following config is for teaching kustomize how to do kustomization for CRDs.
configurations:
- kustomizeconfig.yaml

bases/infrastructure.cluster.x-k8s.io_tinkerbellclusters.yaml under config/crd in https://github.com/tinkerbell/cluster-api-provider-tinkerbell

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.10.0
  creationTimestamp: null
  name: tinkerbellclusters.infrastructure.cluster.x-k8s.io
spec:
  group: infrastructure.cluster.x-k8s.io
  names:
    categories:
    - cluster-api
    kind: TinkerbellCluster
    listKind: TinkerbellClusterList
    plural: tinkerbellclusters
    singular: tinkerbellcluster
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: Cluster to which this TinkerbellCluster belongs
      jsonPath: .metadata.labels.cluster\.x-k8s\.io/cluster-name
      name: Cluster
      type: string
    - description: TinkerbellCluster ready status
      jsonPath: .status.ready
      name: Ready
      type: string
    name: v1beta1
    schema:
      openAPIV3Schema:
        description: TinkerbellCluster is the Schema for the tinkerbellclusters API.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: TinkerbellClusterSpec defines the desired state of TinkerbellCluster.
            properties:
              controlPlaneEndpoint:
                description: "ControlPlaneEndpoint is a required field by ClusterAPI
                  v1beta1. \n See https://cluster-api.sigs.k8s.io/developer/architecture/controllers/cluster.html
                  for more details."
                properties:
                  host:
                    description: The hostname on which the API server is serving.
                    type: string
                  port:
                    description: The port on which the API server is serving.
                    format: int32
                    type: integer
                required:
                - host
                - port
                type: object
              imageLookupBaseRegistry:
                default: ghcr.io/tinkerbell/cluster-api-provider-tinkerbell
                description: ImageLookupBaseRegistry is the base Registry URL that
                  is used for pulling images, if not set, the default will be to use
                  ghcr.io/tinkerbell/cluster-api-provider-tinkerbell.
                type: string
              imageLookupFormat:
                description: 'ImageLookupFormat is the URL naming format to use for
                  machine images when a machine does not specify. When set, this will
                  be used for all cluster machines unless a machine specifies a different
                  ImageLookupFormat. Supports substitutions for {{.BaseRegistry}},
                  {{.OSDistro}}, {{.OSVersion}} and {{.KubernetesVersion}} with the
                  basse URL, OS distribution, OS version, and kubernetes version,
                  respectively. BaseRegistry will be the value in ImageLookupBaseRegistry
                  or ghcr.io/tinkerbell/cluster-api-provider-tinkerbell (the default),
                  OSDistro will be the value in ImageLookupOSDistro or ubuntu (the
                  default), OSVersion will be the value in ImageLookupOSVersion or
                  default based on the OSDistro (if known), and the kubernetes version
                  as defined by the packages produced by kubernetes/release: v1.13.0,
                  v1.12.5-mybuild.1, or v1.17.3. For example, the default image format
                  of {{.BaseRegistry}}/{{.OSDistro}}-{{.OSVersion}}:{{.KubernetesVersion}}.gz
                  will attempt to pull the image from that location. See also: https://golang.org/pkg/text/template/'
                type: string
              imageLookupOSDistro:
                default: ubuntu
                description: ImageLookupOSDistro is the name of the OS distro to use
                  when fetching machine images, if not set it will default to ubuntu.
                type: string
              imageLookupOSVersion:
                description: ImageLookupOSVersion is the version of the OS distribution
                  to use when fetching machine images. If not set it will default
                  based on ImageLookupOSDistro.
                type: string
            type: object
          status:
            description: TinkerbellClusterStatus defines the observed state of TinkerbellCluster.
            properties:
              ready:
                description: Ready denotes that the cluster (infrastructure) is ready.
                type: boolean
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}

Expected output

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
    controller-gen.kubebuilder.io/version: v0.10.0
  labels:
    cluster.x-k8s.io/v1beta1: v1beta1
  name: tinkerbellclusters.infrastructure.cluster.x-k8s.io
spec:
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        caBundle: Cg==
        service:
          name: webhook-service
          namespace: system
          path: /convert
      conversionReviewVersions:
      - v1
      - v1beta1
  group: infrastructure.cluster.x-k8s.io
  names:
    categories:
    - cluster-api
    kind: TinkerbellCluster
    listKind: TinkerbellClusterList
    plural: tinkerbellclusters
    singular: tinkerbellcluster
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: Cluster to which this TinkerbellCluster belongs
      jsonPath: .metadata.labels.cluster\.x-k8s\.io/cluster-name
      name: Cluster
      type: string
    - description: TinkerbellCluster ready status
      jsonPath: .status.ready
      name: Ready
      type: string
    name: v1beta1
    schema:
      openAPIV3Schema:
        description: TinkerbellCluster is the Schema for the tinkerbellclusters API.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: TinkerbellClusterSpec defines the desired state of TinkerbellCluster.
            properties:
              controlPlaneEndpoint:
                description: "ControlPlaneEndpoint is a required field by ClusterAPI
                  v1beta1. \n See https://cluster-api.sigs.k8s.io/developer/architecture/controllers/cluster.html
                  for more details."
                properties:
                  host:
                    description: The hostname on which the API server is serving.
                    type: string
                  port:
                    description: The port on which the API server is serving.
                    format: int32
                    type: integer
                required:
                - host
                - port
                type: object
              imageLookupBaseRegistry:
                default: ghcr.io/tinkerbell/cluster-api-provider-tinkerbell
                description: ImageLookupBaseRegistry is the base Registry URL that
                  is used for pulling images, if not set, the default will be to use
                  ghcr.io/tinkerbell/cluster-api-provider-tinkerbell.
                type: string
              imageLookupFormat:
                description: 'ImageLookupFormat is the URL naming format to use for
                  machine images when a machine does not specify. When set, this will
                  be used for all cluster machines unless a machine specifies a different
                  ImageLookupFormat. Supports substitutions for {{.BaseRegistry}},
                  {{.OSDistro}}, {{.OSVersion}} and {{.KubernetesVersion}} with the
                  basse URL, OS distribution, OS version, and kubernetes version,
                  respectively. BaseRegistry will be the value in ImageLookupBaseRegistry
                  or ghcr.io/tinkerbell/cluster-api-provider-tinkerbell (the default),
                  OSDistro will be the value in ImageLookupOSDistro or ubuntu (the
                  default), OSVersion will be the value in ImageLookupOSVersion or
                  default based on the OSDistro (if known), and the kubernetes version
                  as defined by the packages produced by kubernetes/release: v1.13.0,
                  v1.12.5-mybuild.1, or v1.17.3. For example, the default image format
                  of {{.BaseRegistry}}/{{.OSDistro}}-{{.OSVersion}}:{{.KubernetesVersion}}.gz
                  will attempt to pull the image from that location. See also: https://golang.org/pkg/text/template/'
                type: string
              imageLookupOSDistro:
                default: ubuntu
                description: ImageLookupOSDistro is the name of the OS distro to use
                  when fetching machine images, if not set it will default to ubuntu.
                type: string
              imageLookupOSVersion:
                description: ImageLookupOSVersion is the version of the OS distribution
                  to use when fetching machine images. If not set it will default
                  based on ImageLookupOSDistro.
                type: string
            type: object
          status:
            description: TinkerbellClusterStatus defines the observed state of TinkerbellCluster.
            properties:
              ready:
                description: Ready denotes that the cluster (infrastructure) is ready.
                type: boolean
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}

Actual output

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
    controller-gen.kubebuilder.io/version: v0.10.0
  creationTimestamp: "null"
  labels:
    cluster.x-k8s.io/v1beta1: v1beta1
  name: tinkerbellclusters.infrastructure.cluster.x-k8s.io
spec:
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        caBundle: Cg==
        service:
          name: webhook-service
          namespace: system
          path: /convert
      conversionReviewVersions:
      - v1
      - v1beta1
  group: infrastructure.cluster.x-k8s.io
  names:
    categories:
    - cluster-api
    kind: TinkerbellCluster
    listKind: TinkerbellClusterList
    plural: tinkerbellclusters
    singular: tinkerbellcluster
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: Cluster to which this TinkerbellCluster belongs
      jsonPath: .metadata.labels.cluster\.x-k8s\.io/cluster-name
      name: Cluster
      type: string
    - description: TinkerbellCluster ready status
      jsonPath: .status.ready
      name: Ready
      type: string
    name: v1beta1
    schema:
      openAPIV3Schema:
        description: TinkerbellCluster is the Schema for the tinkerbellclusters API.
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: TinkerbellClusterSpec defines the desired state of TinkerbellCluster.
            properties:
              controlPlaneEndpoint:
                description: "ControlPlaneEndpoint is a required field by ClusterAPI
                  v1beta1. \n See https://cluster-api.sigs.k8s.io/developer/architecture/controllers/cluster.html
                  for more details."
                properties:
                  host:
                    description: The hostname on which the API server is serving.
                    type: string
                  port:
                    description: The port on which the API server is serving.
                    format: int32
                    type: integer
                required:
                - host
                - port
                type: object
              imageLookupBaseRegistry:
                default: ghcr.io/tinkerbell/cluster-api-provider-tinkerbell
                description: ImageLookupBaseRegistry is the base Registry URL that
                  is used for pulling images, if not set, the default will be to use
                  ghcr.io/tinkerbell/cluster-api-provider-tinkerbell.
                type: string
              imageLookupFormat:
                description: 'ImageLookupFormat is the URL naming format to use for
                  machine images when a machine does not specify. When set, this will
                  be used for all cluster machines unless a machine specifies a different
                  ImageLookupFormat. Supports substitutions for {{.BaseRegistry}},
                  {{.OSDistro}}, {{.OSVersion}} and {{.KubernetesVersion}} with the
                  basse URL, OS distribution, OS version, and kubernetes version,
                  respectively. BaseRegistry will be the value in ImageLookupBaseRegistry
                  or ghcr.io/tinkerbell/cluster-api-provider-tinkerbell (the default),
                  OSDistro will be the value in ImageLookupOSDistro or ubuntu (the
                  default), OSVersion will be the value in ImageLookupOSVersion or
                  default based on the OSDistro (if known), and the kubernetes version
                  as defined by the packages produced by kubernetes/release: v1.13.0,
                  v1.12.5-mybuild.1, or v1.17.3. For example, the default image format
                  of {{.BaseRegistry}}/{{.OSDistro}}-{{.OSVersion}}:{{.KubernetesVersion}}.gz
                  will attempt to pull the image from that location. See also: https://golang.org/pkg/text/template/'
                type: string
              imageLookupOSDistro:
                default: ubuntu
                description: ImageLookupOSDistro is the name of the OS distro to use
                  when fetching machine images, if not set it will default to ubuntu.
                type: string
              imageLookupOSVersion:
                description: ImageLookupOSVersion is the version of the OS distribution
                  to use when fetching machine images. If not set it will default
                  based on ImageLookupOSDistro.
                type: string
            type: object
          status:
            description: TinkerbellClusterStatus defines the observed state of TinkerbellCluster.
            properties:
              ready:
                description: Ready denotes that the cluster (infrastructure) is ready.
                type: boolean
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}

Comparing this with the expected output, the only difference is the addition of the creationTimestamp field in metadata, with the value of "null".

Kustomize version

v5.0.0

Operating system

MacOS

[KnVerey]

I see that your original resource for the CRD includes the timestamp field in question, and previous version of Kustomize were stripping it out. The behaviour change comes from a bug we fixed in 5.0, where our Strategic Merge Patch implementation violated the spec by removing null values from original documents (whereas nulls only indicate removals when they are in the patch): #4628.

Please reopen if removing the null from the original document does not solve the problem.

/triage resolved
/close

[k8s-ci-robot]

@KnVerey: Closing this issue.

In response to this:

I see that your original resource for the CRD includes the timestamp field in question, and previous version of Kustomize were stripping it out. The behaviour change comes from a bug we fixed in 5.0, where our Strategic Merge Patch implementation violated the spec by removing null values from original documents (whereas nulls only indicate removals when they are in the patch): #4628.

Please reopen if removing the null from the original document does not solve the problem.

/triage resolved
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[abhay-krishna]

@KnVerey Thanks for looking into this! Yes removing the creationTimestamp from the CRD did solve the issue. I saw the fix in #4890, but shouldn't it still preserve the original type of the object?

[KnVerey]

shouldn't it still preserve the original type of the object?

I agree, but I believe it does? In manual tests, I see either creationTimestamp: "null" or creationTimestamp: null, matching whatever was in the input.

[abhay-krishna]

That's interesting. I observed that even though my original CRD base had creationTimestamp: null (above) but the one after kustomize build had creationTimestamp: "null".

[james-callahan]

I'm getting the same issue: a null is getting transformed by kustomize into the string "null".

I managed to make a minimal replication, it seems to take two patches to trigger:

mycrd.yaml:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: mycrd
  creationTimestamp: null
spec: {}

kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - mycrd.yaml
patches:
  - patch: |
      apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      metadata:
        name: mycrd
      # empty patch
  - patch: |
      apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      metadata:
        name: mycrd
      # empty patch

A kustomize build on that results in:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: "null"
  name: mycrd
spec: {}

Please reopen!