Graceful shutdown

[elliotfehr]

Bug Report

What did you do?

Issued a SIGTERM to the operator-sdk while an ansible-runner was in progress.

What did you expect to see?

The operator-sdk should block shutdown until the graceful shutdown option has been exceeded

What did you see instead? Under which circumstances?

The operator-sdk exits immediately when receiving a shutdown signal.

Environment

Operator type:
/language ansible

Kubernetes cluster type:
vanilla

$ operator-sdk version
be54561

$ go version (if language is Go)
go version go1.15.5 darwin/amd64

$ kubectl version
v1.19.10

Possible Solution

As far as I can tell the controller runtime manager expects runners to block until their execution is complete. Since the operator runs the ansible tasks in a goroutine, the process immediately exits when the controller receives a shutdown signal: https://github.com/operator-framework/operator-sdk/blob/master/internal/ansible/runner/runner.go#L255-L300

We also wait for either the proxy or manager to exit gracefully before shutting down here: https://github.com/operator-framework/operator-sdk/blob/master/internal/cmd/ansible-operator/run/cmd.go#L260-L261 and should likely check to make sure the manager has successfully completed all tasks before shutting down the proxy.

Additional context

Attempted adding in the ability to gracefully shutdown here: #4571

Related to #1843 and #4571

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[elliotfehr]

/remove-lifecycle stale

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[elliotfehr]

/remove-lifecycle rotten

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[elliotfehr]

/remove-lifecycle stale

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.