Trivy-operator scanjob errors

[andriktr]

Hey I have an issue with trivy operator scanjob for several images.
Scanjob failing with the following:

{"level":"error","ts":"2023-06-26T09:01:00Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-operator/scan-vulnerabilityreport-77d66d9c54","container":"clamav-scanner","status.reason":"Error","status.message":"2023-06-26T09:00:57.273Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: context deadline exceeded\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:265\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/reconcile/reconcile.go:103\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"error","ts":"2023-06-26T09:27:03Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-operator/scan-vulnerabilityreport-699f449cc7","container":"clamav-db-mirror","status.reason":"Error","status.message":"2023-06-26T09:26:59.671Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:4c352d8fe39c8ddf3bc8229076b836b1345c39fef063969339b519a7244ae531): walk error: failed to process the file: failed to analyze file: failed to analyze cvdupdate/www/dns.txt: semaphore acquire: context deadline exceeded\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:265\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/reconcile/reconcile.go:103\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"error","ts":"2023-06-26T09:27:03Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-operator/scan-vulnerabilityreport-699f449cc7","container":"clamav-db-mirror-updater","status.reason":"Error","status.message":"2023-06-26T09:26:59.930Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:4c352d8fe39c8ddf3bc8229076b836b1345c39fef063969339b519a7244ae531): walk error: failed to process the file: failed to analyze file: failed to analyze cvdupdate/www/dns.txt: semaphore acquire: context deadline exceeded\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:265\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/reconcile/reconcile.go:103\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}


{"level":"error","ts":"2023-06-26T07:43:34Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-operator/scan-vulnerabilityreport-79f985df5d","container":"awx-rsyslog","status.reason":"Error","status.message":"2023-06-26T07:43:29.633Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: semaphore acquire: context deadline exceeded\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":"2023-06-26T07:43:34Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-operator/scan-vulnerabilityreport-79f985df5d","container":"awx-web","status.reason":"Error","status.message":"2023-06-26T07:43:29.051Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: semaphore acquire: context deadline exceeded\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:254\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/reconcile/reconcile.go:102\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.4/pkg/internal/controller/controller.go:235"}

Environment:

• Trivy-Operator version 0.14.1

What can be the reason and how to fix it ?

Thank you.

[chen-keinan]

@andriktr try extending this helm flag

[chen-keinan]

@andriktr any update on this. ?

[andriktr]

@chen-keinan after timeout increase to 15min errors changed a bit:

{"level":"error","ts":"2023-06-28T20:44:03Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-operator/scan-vulnerabilityreport-699f449cc7","container":"clamav-db-mirror","status.reason":"Error","status.message":"2023-06-28T20:43:59.893Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:4c352d8fe39c8ddf3bc8229076b836b1345c39fef063969339b519a7244ae531): walk error: failed to process the file: failed to analyze file: failed to analyze cvdupdate/www/main-62.cdiff: unable to open cvdupdate/www/main-62.cdiff: failed to open: failed to copy: read tcp 10.162.208.105:44970->10.162.34.20:443: read: connection reset by peer\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:265\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/reconcile/reconcile.go:103\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}


{"level":"error","ts":"2023-06-28T20:09:23Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-operator/scan-vulnerabilityreport-79f985df5d","container":"awx-web","status.reason":"Error","status.message":"2023-06-28T20:09:18.182Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:4c88b9396401fd1ddf96189043bde18cd234d76d42378bb4220fcb841ffe738d): walk error: failed to process the file: failed to analyze file: failed to analyze var/lib/awx/public/static/awx/editor.worker.js.map: unable to open var/lib/awx/public/static/awx/editor.worker.js.map: failed to open: failed to copy: unexpected EOF\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:265\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/reconcile/reconcile.go:103\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"error","ts":"2023-06-28T20:13:23Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-operator/scan-vulnerabilityreport-586995c4f8","container":"awx-rsyslog","status.reason":"Error","status.message":"2023-06-28T20:13:18.660Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:4c88b9396401fd1ddf96189043bde18cd234d76d42378bb4220fcb841ffe738d): walk error: failed to process the file: failed to analyze file: failed to analyze var/lib/awx/public/static/css/main.e706ca3d.css.map: unable to open var/lib/awx/public/static/css/main.e706ca3d.css.map: failed to open: failed to copy: unexpected EOF\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:265\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/reconcile/reconcile.go:103\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"error","ts":"2023-06-28T20:13:23Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-operator/scan-vulnerabilityreport-586995c4f8","container":"awx-task","status.reason":"Error","status.message":"2023-06-28T20:12:59.034Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:4c88b9396401fd1ddf96189043bde18cd234d76d42378bb4220fcb841ffe738d): walk error: failed to process the file: failed to analyze file: failed to analyze var/lib/awx/public/static/awx/json.worker.js.map: unable to open var/lib/awx/public/static/awx/json.worker.js.map: failed to open: failed to copy: unexpected EOF\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:265\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:79\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/reconcile/reconcile.go:103\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}

first error contains scanjob pod ip and private registry ip, but this issue appears only for few specific images. All remaining hundreds of images are in same private registry.

[chen-keinan]

@andriktr this error is in trivy project :

walk error: failed to process the file: failed to analyze file: failed to analyze var/lib/awx/public/static/awx/editor.worker.js.map: unable to open var/lib/awx/public/static/awx/editor.worker.js.map: failed to open: failed to copy: unexpected EOF\n"

do you mind opening an issue there with these errors and we will follow it their also

[chen-keinan]

@andriktr any additional input on this issue ? did you saw my above comment ?

[ilpianista]

Same problem here. Increasing operator.scanJobTimeout and trivy.timeout didn't help.

trivy image <image> works fine.

The image is 3GB big.

[chen-keinan]

Same problem here. Increasing operator.scanJobTimeout and trivy.timeout didn't help.

trivy image <image> works fine.

The image is 3GB big.

@ilpianista @andriktr Any chance you can share the image or equivalent public image with same issue. ? so I'll be able to reproduce the issue

[andriktr]

@chen-keinan I can't share the image as we run it in private repo but I can provide a docker file:

FROM python:alpine3.17

RUN mkdir cvdupdate

RUN addgroup -g 1000 cvdupdate && \
    adduser -D -H -G cvdupdate -u 1000 -h /cvdupdate cvdupdate && \
    chown -R cvdupdate:cvdupdate /cvdupdate

USER cvdupdate
RUN pip3 install cvdupdate --user cvdupdate

WORKDIR /cvdupdate

RUN .local/bin/cvd config set --dbdir /cvdupdate/www

RUN .local/bin/cvd update

ENTRYPOINT [".local/bin/cvd", "serve"]

Container which are build using this DOCKER file are failing with the following error:

{"level":"error","ts":"2023-10-04T06:53:11Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-operator/scan-vulnerabilityreport-699f449cc7","container":"clamav-db-mirror","status.reason":"Error","status.message":"2023-10-04T06:53:03.823Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:4c352d8fe39c8ddf3bc8229076b836b1345c39fef063969339b519a7244ae531): walk error: failed to process the file: failed to analyze file: failed to analyze cvdupdate/www/main.cvd: unable to open cvdupdate/www/main.cvd: failed to open: failed to copy: read tcp 10.162.208.144:35278->10.162.34.20:443: read: connection reset by peer\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:299\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:81\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/reconcile/reconcile.go:111\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:227"}

Other containers (hundreds of them) stored in same private repo are scanned without any issues.

I think that it somehow related to the size of the cvupdate/www/main.cvd file in running container

Also worth to mention that trivy image works fine

On another cluster for same image we receive a bit different error message:

"level":"error","ts":"2023-10-04T06:52:37Z","logger":"reconciler.scan job","msg":"Scan job container","job":"trivy-operator/scan-vulnerabilityreport-84d49db4f","container":"clamav-db-mirror-updater","status.reason":"Error","status.message":"2023-10-04T06:52:33.581Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:4c352d8fe39c8ddf3bc8229076b836b1345c39fef063969339b519a7244ae531): walk error: failed to process the file: failed to analyze file: failed to analyze cvdupdate/www/main.cvd: unable to open cvdupdate/www/main.cvd: failed to open: failed to copy: unexpected EOF\n","stacktrace":"github.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).processFailedScanJob\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:299\ngithub.com/aquasecurity/trivy-operator/pkg/vulnerabilityreport/controller.(*ScanJobController).reconcileJobs.func1\n\t/home/runner/work/trivy-operator/trivy-operator/pkg/vulnerabilityreport/controller/scanjob.go:81\nsigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/reconcile/reconcile.go:111\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:227"

[chen-keinan]

@andriktr I'll try it and update you.

[chen-keinan]

@andriktr any update ?

[andriktr]

@chen-keinan Nothing much to add trivy-operator is running the latest version and for few images issue still exist.

@andriktr I'll try it and update you.

What about you?

[chen-keinan]

@andriktr just an idea which work on a different but similar issue is to set the following flag to false

[chary1112004]

@chen-keinan could you please give which property that you are configuring to false as I click to link then it point to

  # -- severity is a comma separated list of severity levels reported by Trivy.
  severity: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL

Thanks!