panic: runtime error: invalid memory address or nil pointer dereference #180

lhriley · 2023-12-04T23:46:50Z

Your environment

Operator Version: 1.8.0

Connect Server Version: 1.7.2

Kubernetes Version:

❯ kubectl version --output=yaml
clientVersion:
  buildDate: "2023-06-14T09:53:42Z"
  compiler: gc
  gitCommit: 25b4e43193bcda6c7328a6d147b1fb73a33f1598
  gitTreeState: clean
  gitVersion: v1.27.3
  goVersion: go1.20.5
  major: "1"
  minor: "27"
  platform: linux/amd64
kustomizeVersion: v5.0.1
serverVersion:
  buildDate: "2023-07-17T09:25:02Z"
  compiler: gc
  gitCommit: 2ce07bca20da6c3fa49a26fe8e5d53cb8a1e073f
  gitTreeState: clean
  gitVersion: v1.27.3-gke.1700
  goVersion: go1.20.5 X:boringcrypto
  major: "1"
  minor: "27"
  platform: linux/amd64

What happened?

Operator pod panics with the following error after recreating the pod:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x137f5b0]

What did you expect to happen?

The operator doesn't panic.

Steps to reproduce

kubectl delete -n onepassword-connect onepassword-connect-operator-abc123-xyz987

Notes & Logs

❯ kubectl -n onepassword-connect logs -f onepassword-connect-operator-855584b64b-qgh2j
2023-12-04T23:40:27Z    INFO    setup   Operator Version: 1.8.0
2023-12-04T23:40:27Z    INFO    setup   Go Version: go1.20.7
2023-12-04T23:40:27Z    INFO    setup   Go OS/Arch: linux/amd64
2023-12-04T23:40:27Z    INFO    setup   Version of operator-sdk: 1.29.0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x137f5b0]

goroutine 1 [running]:
k8s.io/client-go/discovery.convertAPIResource(...)
        /workspace/vendor/k8s.io/client-go/discovery/aggregated_discovery.go:114
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc00004af78, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
        /workspace/vendor/k8s.io/client-go/discovery/aggregated_discovery.go:95 +0x6f0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc000532390, 0x15}, {0xc00004e980, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
        /workspace/vendor/k8s.io/client-go/discovery/aggregated_discovery.go:49 +0x125
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0xc0001b6c80?)
        /workspace/vendor/k8s.io/client-go/discovery/discovery_client.go:328 +0x3de
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0xc0001b70b0?)
        /workspace/vendor/k8s.io/client-go/discovery/discovery_client.go:203 +0x65
k8s.io/client-go/discovery.ServerGroupsAndResources({0x1afd738, 0xc0000dc810})
        /workspace/vendor/k8s.io/client-go/discovery/discovery_client.go:413 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
        /workspace/vendor/k8s.io/client-go/discovery/discovery_client.go:376 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc0005af0c8)
        /workspace/vendor/k8s.io/client-go/discovery/discovery_client.go:651 +0x71
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?)
        /workspace/vendor/k8s.io/client-go/discovery/discovery_client.go:375 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x1afd738?, 0xc0000dc810?})
        /workspace/vendor/k8s.io/client-go/restmapper/discovery.go:148 +0x42
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper.func1()
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/client/apiutil/dynamicrestmapper.go:94 +0x25
sigs.k8s.io/controller-runtime/pkg/client/apiutil.(*dynamicRESTMapper).setStaticMapper(...)
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/client/apiutil/dynamicrestmapper.go:130
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0xc0001a6360?, {0x0, 0x0, 0xfa3b126c3890da01?})
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/client/apiutil/dynamicrestmapper.go:110 +0x182
sigs.k8s.io/controller-runtime/pkg/cluster.setOptionsDefaults.func1(0xc0001df500?)
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/cluster/cluster.go:217 +0x25
sigs.k8s.io/controller-runtime/pkg/cluster.New(0xc000296b40, {0xc0001b7a70, 0x1, 0x0?})
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/cluster/cluster.go:159 +0x18d
sigs.k8s.io/controller-runtime/pkg/manager.New(_, {0xc0001df500, 0x0, 0x0, {{0x1af9e40, 0xc00040b580}, 0x0}, 0x0, {0x0, 0x0}, ...})
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/manager/manager.go:351 +0xf9
main.main()
        /workspace/main.go:144 +0x535

The text was updated successfully, but these errors were encountered:

lhriley · 2023-12-05T01:29:33Z

I did some digging and this looks like a problem that would be resolved by updating the dependencies to a version that supports a modern kubernetes version.

I briefly tried to upgrade dependencies to test, however, there are several breaking changes due to deprecations in the package raising the errors.

Since this hasn't seen a new build in 4 months, it is critical that this application is assigned to someone for ongoing maintenance considering how quickly kubernetes moves.

Related:
kubernetes-sigs/controller-runtime#2422

Pinned version: https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.14.5

Breaking versions:

latest version: https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.16.3

There is no compatible release in the v0,14.x tree.

edif2008 · 2023-12-18T13:05:58Z

Hey @lhriley! 👋🏻

Thank you for raising this with us and providing so many details. This will help us in narrowing down the problem and find out a solution to this.
We're currently investigating this and we will come back to you when we have any updates. 😄

edif2008 · 2024-01-02T16:41:31Z

Hey @lhriley,

We've got the release notes on the latest operator-sdk v1.33.0, which provide some migration steps to upgrade some dependencies and scaffolding of this project. This release came out right after we've replied to you.

This migration also addresses the issue you're facing, as well as updating the project's dependencies to their latest versions.
I'll be working on this migration for the upcoming weeks and will reach out to you once we've reached to a good state with it. 😄

lhriley · 2024-01-02T20:29:31Z

@edif2008 thanks for the update. Looking forward to a new release!

rstreics-lb · 2024-01-30T10:57:14Z

@edif2008 Hey we rolled out latest version and but still fails retrieve 1P records

2024-01-30T10:46:03Z    INFO    setup   Operator Version: 1.8.1
2024-01-30T10:46:03Z    INFO    setup   Go Version: go1.21.6
2024-01-30T10:46:03Z    INFO    setup   Go OS/Arch: linux/amd64
2024-01-30T10:46:03Z    INFO    setup   Version of operator-sdk: 1.33.0
2024-01-30T10:46:03Z    INFO    setup   Automated Connect Management Disabled
2024-01-30T10:46:03Z    INFO    setup   starting manager
2024-01-30T10:46:03Z    INFO    controller-runtime.metrics      Starting metrics server
2024-01-30T10:46:03Z    INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8080", "secure": false}
2024-01-30T10:46:03Z    INFO    starting server {"kind": "health probe", "addr": "[::]:8081"}
2024-01-30T10:46:03Z    INFO    Starting EventSource    {"controller": "onepassworditem", "controllerGroup": "onepassword.com", "controllerKind": "OnePasswordItem", "source": "kind source: *v1.OnePasswordItem"}
2024-01-30T10:46:03Z    INFO    Starting Controller     {"controller": "onepassworditem", "controllerGroup": "onepassword.com", "controllerKind": "OnePasswordItem"}
2024-01-30T10:46:03Z    INFO    Starting EventSource    {"controller": "deployment", "controllerGroup": "apps", "controllerKind": "Deployment", "source": "kind source: *v1.Deployment"}
2024-01-30T10:46:03Z    INFO    Starting Controller     {"controller": "deployment", "controllerGroup": "apps", "controllerKind": "Deployment"}
2024-01-30T10:46:03Z    INFO    Starting workers        {"controller": "onepassworditem", "controllerGroup": "onepassword.com", "controllerKind": "OnePasswordItem", "worker count": 1}
2024-01-30T10:46:03Z    INFO    Starting workers        {"controller": "deployment", "controllerGroup": "apps", "controllerKind": "Deployment", "worker count": 1}
2024-01-30T10:46:03Z    ERROR   Reconciler error        {"controller": "onepassworditem", "controllerGroup": "onepassword.com", "controllerKind": "OnePasswordItem", "OnePasswordItem": {"name":"*****,"namespace":"default"}, "namespace": "default", "name": "******", "reconcileID": "8e2eb6fe-4d21-43a4-a05b-3771ea6ce9f8", "error": "Failed to retrieve item: status 401: Invalid bearer token"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-01-30T10:46:03Z    ERROR   Reconciler error        {"controller": "onepassworditem", "controllerGroup": "onepassword.com", "controllerKind": "OnePasswordItem", "OnePasswordItem": {"name":"********","namespace":"default"}, "namespace": "default", "name": "****", "reconcileID": "628a8ea9-53d6-4f5f-a473-9b7169da9b1d", "error": "Failed to retrieve item: status 401: Invalid bearer token"}

edif2008 · 2024-01-30T11:15:10Z

Hey @rstreics-lb,

It seems that now the issue is different:

Failed to retrieve item: status 401: Invalid bearer token

This sounds like the Connect token the operator uses is invalid. 🤔

lhriley added the bug Something isn't working label Dec 4, 2023

edif2008 self-assigned this Jan 2, 2024

edif2008 mentioned this issue Jan 5, 2024

Upgrade the operator to use Operator SDK v1.33.0 #182

Merged

edif2008 closed this as completed in #182 Jan 25, 2024

github-actions bot mentioned this issue Jan 25, 2024

Prepare Release - v1.8.1 #183

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

panic: runtime error: invalid memory address or nil pointer dereference #180

panic: runtime error: invalid memory address or nil pointer dereference #180

lhriley commented Dec 4, 2023

lhriley commented Dec 5, 2023 •

edited

Loading

Uh oh!

edif2008 commented Dec 18, 2023

Uh oh!

edif2008 commented Jan 2, 2024

Uh oh!

lhriley commented Jan 2, 2024

Uh oh!

rstreics-lb commented Jan 30, 2024 •

edited

Loading

Uh oh!

edif2008 commented Jan 30, 2024 •

edited

Loading

Uh oh!

panic: runtime error: invalid memory address or nil pointer dereference #180

panic: runtime error: invalid memory address or nil pointer dereference #180

Comments

lhriley commented Dec 4, 2023

Your environment

What happened?

What did you expect to happen?

Steps to reproduce

Notes & Logs

lhriley commented Dec 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edif2008 commented Dec 18, 2023

Uh oh!

edif2008 commented Jan 2, 2024

Uh oh!

lhriley commented Jan 2, 2024

Uh oh!

rstreics-lb commented Jan 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edif2008 commented Jan 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lhriley commented Dec 5, 2023 •

edited

Loading

rstreics-lb commented Jan 30, 2024 •

edited

Loading

edif2008 commented Jan 30, 2024 •

edited

Loading