[DISCO-2169] Delete CircleCI approval task for prod deployment

[Trinaa]

References

JIRA: DISCO-2169
GitHub: N/A

Description

Remove the 'unhold-to-deploy-to-prod' CircleCI step in order to fulfill Rapid Release.

PR Review Checklist

Put an x in the boxes that apply

• This PR conforms to the Contribution Guidelines
• The PR title starts with the JIRA issue reference, format [DISCO-####], and has the same title (if applicable)
• [do not deploy] and [load test: (abort|warn)] keywords are applied (if applicable)
• Documentation has been updated (if applicable)
• Functional and performance test coverage has been expanded and maintained (if applicable)

[Trinaa]

@ncloudioj & @hackebrot
Does this change fulfill the task? Do we need to sync with SRE on anything?

[hackebrot]

Hi @Trinaa! 👋🏻

It would be good to sync with Services SRE on this change, yes.

We definitely also want to update any documentation that references the CircleCI approval task and ensure that everybody on the team merging pull requests to the main branch is aware that this triggers a prod deployment and requires active monitoring.

[ncloudioj]

Let's also update the deployment doc to reflect this change.

[hackebrot]

LGTM! 👍🏻

[hackebrot]

Please don't merge this pull request just yet, @Trinaa. ⚠️

The changes look good to me, but I'd like to make sure everyone is aware of this change to the release process.

[mtbottle]

As I understand, docker-image-publish-prod does a deploy to both staging and prod and docker-image-publish-staging also does a deploy to staging. I don't think it makes sense to have 2 staging deploys, so I would vote to remove one of them.

[ncloudioj]

As I understand, docker-image-publish-prod does a deploy to both staging and prod and docker-image-publish-staging also does a deploy to staging. I don't think it makes sense to have 2 staging deploys, so I would vote to remove one of them.

Good call out. Yup, I think we can drop the second stage deployment from the pipeline. This needs SRE's help for reconfiguration.

[Trinaa]

As I understand, docker-image-publish-prod does a deploy to both staging and prod and docker-image-publish-staging also does a deploy to staging. I don't think it makes sense to have 2 staging deploys, so I would vote to remove one of them.

Good call out. Yup, I think we can drop the second stage deployment from the pipeline. This needs SRE's help for reconfiguration.

Spoke to @dlactin about this earlier, he is already on it 🚀

[Trinaa]

@hackebrot @ncloudioj
Following SRE's recent change, disabling Docker image validation in Merino-py, I am able to set the creation and publication of the load test Docker image as a dependency to publishing the stage and production Merino Docker images.

I have also consolidated the docker-image-publish-stage and docker-image-publish-prod steps.

Please have a look

Proposed new CircleCI flow:

[ncloudioj]

🚢

[hackebrot]

Great work, @Trinaa! 🚀

[quiiver]

AFAICT, there is no stage health check required before deploying to production. Is that the case? If so, it concerns me that we could deploy to prod with staging broken.

[ncloudioj]

AFAICT, there is no stage health check required before deploying to production. Is that the case? If so, it concerns me that we could deploy to prod with staging broken.

Good point, @quiiver. We currently only have a health check against prod-canary, if the check fails, the production deployment will abort. @Trinaa filed this followup ticket (DISCO-2322) to add some basic smoke tests for staging. I think we can merge this as is and keep a closer eye on how it goes in the wild.

Does that make sense?

[Trinaa]

Follow-up: DISCO-2322 Add smoke tests to Merino CD pipeline.

Also looking at the merino Jenkinsfile-py I think the canary also applies to stage.