Skip to content

pkg/scan and the web UI report every coverage loss as "no document text extracted", including losses where the text WAS read #432

Closed

Description

@sectoramen

Summary

internal/core/scanner.go labels every EmptyExtractionFiles entry as "no document text
extracted from", regardless of why coverage was lost. The CLI was fixed for exactly this
mislabelling; the library and web surfaces were not, so they still report a true fact under a false
heading.

internal/core/scanner.go:222:

incompleteReason = fmt.Sprintf("no document text extracted from %s: %s",
    stats.EmptyExtractionFiles[0].FilePath, stats.EmptyExtractionFiles[0].Reason)

That string is the only description these consumers get:

  • pkg/scan.ScanResult.IncompleteReason (pkg/scan/scan.go:215, set at :329)
  • the web UI's incomplete_reason field (internal/web/server.go:98, set at :589, rendered via
    :792)

Why it is wrong

The ExtractionWarning channel carries two different statements, and only one of them is "the body
held no text":

producer what actually happened
office text extractor, no body part the body really was empty — no text is correct
office metadata, embedded part over the cap the body WAS read and scanned; one part was not
audio, malformed WAV chunk layout metadata WAS recovered; the walk was realigned
video, box walk stopped early (#398) some boxes WERE read; the rest were not

For every row but the first, "no document text extracted" asserts a failure that did not happen. An
operator reading it concludes the document was unreadable, when in fact it was scanned and one part
of it was skipped — which is a different action to take.

cmd/unscanned_report.go:270-285 already draws this distinction, bucketing on the presence of
noBodyTextPrefix and reporting coverage cut short versus no body text. cmd/extraction_warning_cause_test.go
holds a table of every real producer's wording to keep new producers in the right bucket. None of
that reaches the library path, which does its own formatting.

Reproduction

Any file that produces a cut-short warning, scanned through pkg/scan or the web UI rather than the
CLI. A video whose box walk stops early is the newest example; an Office document with an
over-cap embedded part is the oldest.

Suggested direction

The substring test is a workaround for a missing field. EmptyExtractionFiles[] entries would be
better carrying a structured cause set by the producer, which both cmd and core then render.
That removes the phrasing dependency entirely — today a producer that happens to write "no text
extracted from" in a cut-short warning is silently filed under the wrong cause, and the only thing
preventing it is a test table someone has to remember to extend.

Failing that, core.ScanContent should call the same classifier cmd uses and pick its wording from
the result.

Notes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions