Skip to content

Impossible to redeploy a non blue-green application #367

Open
Open
Impossible to redeploy a non blue-green application#367
Labels
bug

Description

@lunika
lunika
opened on Sep 16, 2019

Bug Report

Expected behavior/code

When I deploy an application with blue-green settings set to false and parameters changed in a template (DeploymentConfig mostly), I expect a to deploy without error with a rolling-update strategy

Actual Behavior

The deploy fail with this kind of error:

TASK [OpenShift deployments with deployment_stamp[d-190916-09h04m48s] must be present] *******************************************************************************************************
failed: [local] (item=apps/learninglocker/templates/services/app/dc_api.yml.j2) => {"changed": false, "error": 422, "item": "apps/learninglocker/templates/services/app/dc_api.yml.j2", "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"DeploymentConfig \\\\\"learninglocker-api\\\\\" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{\\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-api\\\\\", \\\\\"service\\\\\":\\\\\"api\\\\\", \\\\\"version\\\\\":\\\\\"v2.8.2\\\\\"}: `selector` does not match template `labels`\",\"reason\":\"Invalid\",\"details\":{\"name\":\"learninglocker-api\",\"kind\":\"DeploymentConfig\",\"causes\":[{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid value: map[string]string{\\\\\"version\\\\\":\\\\\"v2.8.2\\\\\", \\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-api\\\\\", \\\\\"service\\\\\":\\\\\"api\\\\\"}: `selector` does not match template `labels`\",\"field\":\"spec.template.metadata.labels\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
failed: [local] (item=apps/learninglocker/templates/services/app/dc_ui.yml.j2) => {"changed": false, "error": 422, "item": "apps/learninglocker/templates/services/app/dc_ui.yml.j2", "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"DeploymentConfig \\\\\"learninglocker-ui\\\\\" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{\\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-ui\\\\\", \\\\\"service\\\\\":\\\\\"ui\\\\\", \\\\\"version\\\\\":\\\\\"v2.8.2\\\\\"}: `selector` does not match template `labels`\",\"reason\":\"Invalid\",\"details\":{\"name\":\"learninglocker-ui\",\"kind\":\"DeploymentConfig\",\"causes\":[{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid value: map[string]string{\\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-ui\\\\\", \\\\\"service\\\\\":\\\\\"ui\\\\\", \\\\\"version\\\\\":\\\\\"v2.8.2\\\\\"}: `selector` does not match template `labels`\",\"field\":\"spec.template.metadata.labels\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
failed: [local] (item=apps/learninglocker/templates/services/app/dc_worker.yml.j2) => {"changed": false, "error": 422, "item": "apps/learninglocker/templates/services/app/dc_worker.yml.j2", "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"DeploymentConfig \\\\\"learninglocker-worker\\\\\" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{\\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-worker\\\\\", \\\\\"service\\\\\":\\\\\"worker\\\\\", \\\\\"version\\\\\":\\\\\"v2.8.2\\\\\"}: `selector` does not match template `labels`\",\"reason\":\"Invalid\",\"details\":{\"name\":\"learninglocker-worker\",\"kind\":\"DeploymentConfig\",\"causes\":[{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid value: map[string]string{\\\\\"service\\\\\":\\\\\"worker\\\\\", \\\\\"version\\\\\":\\\\\"v2.8.2\\\\\", \\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-worker\\\\\"}: `selector` does not match template `labels`\",\"field\":\"spec.template.metadata.labels\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}
failed: [local] (item=apps/learninglocker/templates/services/xapi/dc.yml.j2) => {"changed": false, "error": 422, "item": "apps/learninglocker/templates/services/xapi/dc.yml.j2", "msg": "Failed to patch object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"DeploymentConfig \\\\\"learninglocker-xapi\\\\\" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{\\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-xapi\\\\\", \\\\\"service\\\\\":\\\\\"xapi\\\\\", \\\\\"version\\\\\":\\\\\"v2.9.2\\\\\"}: `selector` does not match template `labels`\",\"reason\":\"Invalid\",\"details\":{\"name\":\"learninglocker-xapi\",\"kind\":\"DeploymentConfig\",\"causes\":[{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid value: map[string]string{\\\\\"version\\\\\":\\\\\"v2.9.2\\\\\", \\\\\"app\\\\\":\\\\\"learninglocker\\\\\", \\\\\"deploymentconfig\\\\\":\\\\\"learninglocker-xapi\\\\\", \\\\\"service\\\\\":\\\\\"xapi\\\\\"}: `selector` does not match template `labels`\",\"field\":\"spec.template.metadata.labels\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}

Steps to Reproduce

  1. Deploy a first version of a non blue-green compatible application (like learning locker)
  2. Change the image tag version
  3. Deploy it again

Environment

  • Arnold version: 2.7.0 and master
  • Platform: our containers.

Possible Solution

I tried to use the force parameter in the step OpenShift deployments with deployment_stamp[{{ deployment_stamp }}] must be {{ deployment_state | default('present') }} in manage_app.yml task but I have a new error I don't really understand:

failed: [local] (item=apps/learninglocker/templates/services/app/dc_api.yml.j2) => {"changed": false, "error": 422, "item": "apps/learninglocker/templates/services/app/dc_api.yml.j2", "msg": "Failed to replace object: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"deploymentconfigs \\\\\"learninglocker-api\\\\\" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update\",\"reason\":\"Invalid\",\"details\":{\"name\":\"learninglocker-api\",\"kind\":\"deploymentconfigs\",\"causes\":[{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid value: 0x0: must be specified for an update\",\"field\":\"metadata.resourceVersion\"}]},\"code\":422}\\n'", "reason": "Unprocessable Entity", "status": 422}

Activity

  1. lunika
    added
    bug
    on Sep 16, 2019

  2. madmatah commented on Jan 23, 2020

    @madmatah
    madmatah
    on Jan 23, 2020
    Contributor

    I've just got the same issue with a non blue/green application.

    My DeploymentConfig definition was something like this :

    apiVersion: v1
kind: DeploymentConfig
metadata:
  labels:
    app: elasticsearch
    service: elasticsearch
    version: "{{ elasticsearch_image_tag }}"
    type: es-node
  name: "elasticsearch-node"
  namespace: "{{ project_name }}"
spec:
  replicas: {{ elasticsearch_nodes }}
  template:
    metadata:
      labels:
        app: elasticsearch
        service: elasticsearch
        version: "{{ elasticsearch_image_tag }}"
        type: es-node
    spec:
      containers:
        - image: {{ elasticsearch_image_name }}:{{ elasticsearch_image_tag }}
          name: elasticsearch

    It seems that once a DeploymentConfig resource is deployed, it cannot be patched if the labels defined in spec.template.metadata.labels change. They are used as a label selector to target the resource to patch.

    I removed the dynamic parts this section and it worked ! 🎉

    apiVersion: v1
kind: DeploymentConfig
metadata:
  labels:
    app: elasticsearch
    service: elasticsearch
    version: "{{ elasticsearch_image_tag }}"
    type: es-node
  name: "elasticsearch-node"
  namespace: "{{ project_name }}"
spec:
  replicas: {{ elasticsearch_nodes }}
  template:
    metadata:
      labels:
        app: elasticsearch
        service: elasticsearch
        type: es-node
    spec:
      containers:
        - image: {{ elasticsearch_image_name }}:{{ elasticsearch_image_tag }}
          name: elasticsearch

    Once deployed and patched :

    $> oc get dc --show-labels=true
NAME                 REVISION   DESIRED   CURRENT   TRIGGERED BY   LABELS
elasticsearch-node   2          3         3         config         app=elasticsearch,service=elasticsearch,type=es-node,version=6.3.1

  3. jmaupetit commented on Jan 23, 2020

    @jmaupetit
    jmaupetit
    on Jan 23, 2020
    Contributor

    What if we use the force: true flag during DC creation/patching?

  4. madmatah commented on Jan 23, 2020

    @madmatah
    madmatah
    on Jan 23, 2020
    Contributor

    What if we use the force: true flag during DC creation/patching?

    It doesn't work (see @lunika's comment in Possible solution)
    I also tried merge_type: merge, merge_type: strategic-merge and apply: yes.

  5. jmaupetit commented on Jan 23, 2020

    @jmaupetit
    jmaupetit
    on Jan 23, 2020
    Contributor

    So, in case of version upgrade, should we delete and then create the object instead of patching it?

  6. madmatah commented on Jan 23, 2020

    @madmatah
    madmatah
    on Jan 23, 2020
    Contributor

    If we do it that way (delete / create), all interesting features of the DeploymentConfig (like rollback or deployment strategies) will not be usable.

    What do you think about the solution i proposed ?
    Dynamic labels are still usable for us and re-deployment is working.

  7. jmaupetit commented on Jan 23, 2020

    @jmaupetit
    jmaupetit
    on Jan 23, 2020
    Contributor

    If thought it was a test to understand the issue, not a solution per se 😅

    So if we follow your recommendation, we will loose version labels in pods, right? Knowing that the information is in the image tag, I think it's a reasonable approach.

  8. madmatah commented on Jan 27, 2020

    @madmatah
    madmatah
    on Jan 27, 2020
    Contributor

    You're right, we lose the version label in pods but we keep it in deploymentconfig.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions