fix: enable all template types for template list/display

[dwisiswant0]

Proposed changes

fix(runner): enable all template types for template list/display

When using -tl or -td, explicitly enable all
template types (code, file, self-contained,
global-matchers, headless, dast) to ensure
all templates are listed w/o requiring extra flags.

Fixes #6644.

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Summary by CodeRabbit

• New Features

  • Templates now declare explicit capability requirements and expose richer per-protocol predicates and helpers for enablement and fuzzing detection.
  • Request types add a HasFuzzing indicator.

• Bug Fixes

  • Duplicate templates are detected and skipped earlier to prevent repeated entries.

• Chores

  • Removed prior universal template enabling and introduced centralized gating; deprecated some legacy helpers.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

Centralizes template enablement using new Capabilities + Template.IsEnabledFor, removes runner's early universal enabling, adds many Template HasXRequest/HasWorkflows predicates and fuzzing accessors, and introduces pre-append duplicate detection in the loader.

Changes

Cohort / File(s)	Change Summary
Runner: remove early universal enabling internal/runner/runner.go	Removes the New-path block that forced EnableCodeTemplates, EnableFileTemplates, EnableSelfContainedTemplates, and EnableGlobalMatchersTemplates when TemplateList or TemplateDisplay were set.
Loader: unified gating & duplicate guard pkg/catalog/loader/loader.go	Derives caps (Capabilities) and isListOrDisplay from executor options; gates templates via template.IsEnabledFor(caps) (unless listing/displaying); adds pre-append duplicate ID detection and nil-template guards; consolidates ExecutorOptions field usage through a local alias.
Templates core: requirements & capabilities pkg/templates/templates.go	Adds Requirements and Capabilities types and Template.Requirements() / Template.IsEnabledFor(caps Capabilities); introduces deprecated helpers and replaces many length checks with new predicate usage in type/enablement logic.
Templates utils: request predicates & helpers pkg/templates/templates_utils.go	Adds generic HasRequest[T protocols.Request](...) and many Template.HasXRequest predicates (DNS, File, HTTP, Headless, Network, SSL, Websocket, WHOIS, Code, Javascript), plus HasQueueRequests, HasWorkflows, IsFuzzableRequest, IsFlowTemplate, IsGlobalMatchersTemplate.
Templates compile/workflows: predicate adoption pkg/templates/compile.go, pkg/templates/workflows.go	Replaces direct slice-length checks with HasXRequest predicates (including headless requiring global headless option); updates code workflow checks to use HasCodeRequest().
Protocols: expose fuzzing checks pkg/protocols/headless/headless.go, pkg/protocols/http/http.go	Adds HasFuzzing() bool methods on Headless and HTTP Request types to expose whether fuzzing rules exist.
Manifest go.mod	Module manifest present (no functional change described).

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant Runner
  participant Loader
  participant Template
  participant Store

  Runner->>Loader: create loader with executor options
  Loader->>Loader: derive caps (Capabilities) and isListOrDisplay
  loop each candidate template
    Loader->>Template: parse/load template
    alt template == nil
      Loader-->>Loader: skip
    else
      alt isListOrDisplay
        Loader-->>Loader: bypass IsEnabledFor gating
      else
        Loader->>Template: IsEnabledFor(caps)?
        Template-->>Loader: enabled/disabled
      end
      alt ID already seen
        Loader-->>Loader: log & skip (duplicate)
      else
        Loader->>Template: set Path
        Loader->>Store: append to store.templates
      end
    end
  end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

• Inspect pkg/templates Requirements/Capabilities and each HasXRequest predicate for correctness, nil-safety, and deprecation annotations.
• Verify pkg/catalog/loader/loader.go: gating logic, isListOrDisplay bypass, duplicate-ID detection, option aliasing, and interactions with author/tag filters.
• Confirm removal in internal/runner/runner.go doesn't leave unmet initialization assumptions elsewhere.
• Review HasFuzzing() additions in protocol packages for API consistency.

Poem

🐰 I hopped through templates, caps in paw,

I checked each request, no duplicate flaw,
Paths were set gently, gates neatly met,
Fuzzing whispers answered — no fret,
A tiny hop, a tidy loader law 🥕

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 62.50% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: enabling all template types when using template list/display commands.
Linked Issues check	✅ Passed	The PR addresses the core requirement from #6644 by enabling all template types (code, file, self-contained, global-matchers, headless, dast) for template list/display operations to ensure accurate template counts and listings.
Out of Scope Changes check	✅ Passed	The changes are scoped to the template loading and gating logic necessary to enable all template types for list/display operations, with supporting refactoring of predicate methods.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch dwisiswant0/fix/runner/enable-all-template-types-for-template-list-display

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 56b6d42 and 8fd15b0.

📒 Files selected for processing (1)

• internal/runner/runner.go (1 hunks)

🧰 Additional context used

📓 Path-based instructions (2)

**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.go: Format Go code using go fmt ./...
Run static analysis using go vet ./... on Go code

Files:

• internal/runner/runner.go

internal/runner/**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

Core scanning orchestration logic should be implemented in internal/runner

Files:

• internal/runner/runner.go

🧬 Code graph analysis (1)

internal/runner/runner.go (1)

lib/config.go (4)

• EnableCodeTemplates (394-400)
• EnableFileTemplates (427-432)
• EnableSelfContainedTemplates (403-408)
• EnableGlobalMatchersTemplates (411-416)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: Lint

[Mzack9999]

proposed minor change

[dwisiswant0]

Drafting. I'll do more cleanup.

[dwisiswant0]

tbh I think we've reached the stage where having a GLOSARRY would help a lot. We should probably start drafting one so everyone stays on the same page.

[dwisiswant0]

Introduced in #6343. Removed, since mutating global opts in the runner could affects the entire engine state (UB).

[coderabbitai]

Actionable comments posted: 0

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

pkg/catalog/loader/loader.go (1)

187-221: Fix error handling when remote templates list is empty but no error occurred

In ReadTemplateFromURI, the condition if err != nil || len(remoteTemplates) == 0 can result in calling errkit.Wrapf(nil, ...) when getRemoteTemplatesAndWorkflows returns an empty slice with err == nil. Since Wrapf returns nil when wrapping a nil error, this causes the function to return (nil, nil) instead of a proper error.

Separate the error cases:

if err != nil {
    return nil, errkit.Wrapf(err, "Could not load template %s", uri)
}
if len(remoteTemplates) == 0 {
    return nil, fmt.Errorf("Could not load template %s: empty response", uri)
}

♻️ Duplicate comments (1)

pkg/catalog/loader/loader.go (1)

684-703: Executor options alias and dialer lookup are fine; dialer panic is already tracked separately

Aliasing typesOpts := store.config.ExecutorOptions.Options improves readability, and lazily initializing ExecutionId when empty is reasonable.

The subsequent dialer lookup:

dialers := protocolstate.GetDialersWithId(typesOpts.ExecutionId)
if dialers == nil {
    panic("dialers with executionId " + typesOpts.ExecutionId + " not found")
}

preserves the existing “hard fail” behaviour when dialers are missing. This is the same concern previously discussed and tracked in issue #6674, so no change is required in this PR, but it’s worth keeping in mind for future error-handling improvements.

If you want to revisit it in a separate PR, the existing issue already contains the suggested refactor to convert this panic into a proper error return.

🧹 Nitpick comments (5)

pkg/catalog/loader/loader.go (2)

410-477: Capabilities-based gating plus dedupe in LoadTemplatesOnlyMetadata looks correct for -tl/-td

The new caps := templates.Capabilities{...} plus template.IsEnabledFor(caps) gate, combined with the isListOrDisplay short-circuit, achieves the right balance:

• For normal runs, templates whose requirements (headless/code/DAST/self-contained/file) aren’t satisfied by options are filtered out early at metadata time.
• For template list/display (TemplateList or TemplateDisplay), isListOrDisplay disables this gating so all template types are included regardless of flags, matching the PR objective.
• Pre-append de‑dupe via loadedTemplateIDs and setting template.Path before appending keeps store.templates clean and deterministic.

This aligns the metadata-only path with the new Requirements/Capabilities model and should avoid the previous under-counting for -tl/-td + filters.

As a follow-up (not for this PR), you might consider reusing IsEnabledFor in the main LoadTemplatesWithTags path too, to reduce the chance of drift between metadata and full-parse gating.

---

751-808: Gating chain for unsigned/self-contained/file/headless/code/DAST templates is coherent with new helpers

The updated gating logic in LoadTemplatesWithTags:

• Uses typesOpts consistently (avoid repeated store.config.ExecutorOptions.Options).
• Applies unsigned, self-contained, and file gating before protocol/DAST logic.
• For DAST mode (typesOpts.DAST && store.ID() != AuthStoreId), loads only fuzzable or global matchers templates, with a headless flag check.
• Outside DAST mode, enforces:
  • Headless flag for any headless requests (parsed.HasHeadlessRequest()),
  • EnableCodeTemplates flag for code templates,
  • Code-template verification (skipping unsigned code templates),
  • Excluding DAST-only templates unless -dast is set.

All of this now relies on the predicate helpers (HasHeadlessRequest, HasCodeRequest, IsFuzzableRequest, IsGlobalMatchersTemplate, HasWorkflows), which centralizes protocol detection and should reduce subtle length-check bugs. Behaviourally this aligns with prior intent, while making the DAST / headless / code gating easier to reason about.

Long term, you could consider expressing more of this logic in terms of Template.Requirements() + IsEnabledFor, with an explicit “DAST-only view” instead of duplicating predicate checks here.

pkg/templates/templates.go (3)

166-179: Deprecated HasCodeProtocol/HasFileProtocol shims are a reasonable compatibility bridge

Exposing HasCodeProtocol and HasFileProtocol as thin, deprecated wrappers around the underlying slices keeps existing callers compiling while steering new code toward the HasCodeRequest/HasFileRequest predicate family. This is a pragmatic way to evolve the API.

Documenting planned removal (e.g., in a changelog or deprecation policy) will help downstream users schedule migrations off these helpers.

---

181-207: Centralizing protocol presence and request-ID logic on HasXRequest/HasRequest is sound

The changes in Type, validateAllRequestIDs, and UnmarshalYAML consistently move from manual slice length checks to:

• HasXRequest() / HasXRequest(min) methods on Template.
• Generic HasRequest(...) checks for HTTP vs http / network vs tcp alias fields.

This yields a few benefits:

• Single-point logic for “does this template have protocol X?” including future protocol extensions.
• Safer multi-protocol detection via hasMultipleRequests() and YAML key order preservation (addRequestsToQueue).
• Clearer conflict errors when both legacy and new fields are present (e.g., http and requests together).

The ordering in Type() still matches the old implicit priority (DNS, File, HTTP, ... , Javascript, Workflows), so multi-protocol templates preserve their principal type.

You may eventually want to base hasMultipleRequests() on the same HasXRequest helpers to avoid divergence with any future protocol-counting logic, but it’s fine as-is.

Also applies to: 243-325, 335-405

---

592-647: Requirements / Capabilities and IsEnabledFor form a solid basis for unified gating

The new model:

• Requirements encodes what a template needs to run:
  • Headless: any headless requests.
  • Code: any code requests.
  • DAST: any fuzzable requests (via IsFuzzableRequest()).
  • SelfContained: template-level self-contained flag.
  • File: any file requests.
• Capabilities mirrors this shape from execution options.
• IsEnabledFor simply checks that each true requirement is satisfied by the corresponding capability.

This is simple, composable, and matches how LoadTemplatesOnlyMetadata builds caps. It should make future gating changes easier (e.g., adding new capability bits) without scattering checks across the codebase.

Over time, consolidating more of the loader’s per-flag gating into IsEnabledFor (with a DAST-specific view where needed) would further reduce duplicated logic.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8a6e9c8 and 2fc2ed6.

📒 Files selected for processing (7)

• pkg/catalog/loader/loader.go (17 hunks)
• pkg/protocols/headless/headless.go (1 hunks)
• pkg/protocols/http/http.go (1 hunks)
• pkg/templates/compile.go (1 hunks)
• pkg/templates/templates.go (7 hunks)
• pkg/templates/templates_utils.go (1 hunks)
• pkg/templates/workflows.go (1 hunks)

🚧 Files skipped from review as they are similar to previous changes (3)

• pkg/protocols/headless/headless.go
• pkg/protocols/http/http.go
• pkg/templates/templates_utils.go

🧰 Additional context used

📓 Path-based instructions (3)

**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.go: Format Go code using go fmt ./...
Run static analysis using go vet ./... on Go code

Files:

• pkg/catalog/loader/loader.go
• pkg/templates/compile.go
• pkg/templates/templates.go
• pkg/templates/workflows.go

pkg/catalog/**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

Template discovery and loading from disk/remote sources should be implemented in pkg/catalog

Files:

• pkg/catalog/loader/loader.go

pkg/templates/**/*.go

📄 CodeRabbit inference engine (CLAUDE.md)

Template parsing, compilation, and management should be implemented in pkg/templates

Files:

• pkg/templates/compile.go
• pkg/templates/templates.go
• pkg/templates/workflows.go

🧬 Code graph analysis (1)

pkg/catalog/loader/loader.go (3)

pkg/templates/templates.go (2)

• Capabilities (613-619)
• Template (35-164)

pkg/types/types.go (2)

• Options (34-472)
• DefaultTemplateLoadingConcurrency (23-23)

pkg/protocols/common/protocolstate/state.go (1)

• GetDialersWithId (40-46)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: Lint

🔇 Additional comments (6)

pkg/templates/workflows.go (1)

95-111: Using HasCodeRequest() here is consistent with new request predicates

Switching from a raw slice length check to template.HasCodeRequest() keeps behaviour the same while centralizing protocol detection logic; the workflow gating for code templates (flag + verification) remains intact and looks correct.

pkg/catalog/loader/loader.go (3)

630-647: LoadWorkflows behaviour unchanged and still straightforward

The workflow loading path remains simple: catalog resolution, LoadWorkflow, and then templates.Parse with warnings on errors. The surrounding changes don’t alter semantics; appending only parsed, non-nil workflows is still correct.

---

842-853: HTTP-based protocol detection via HasHTTPRequest / HasHeadlessRequest is clearer

IsHTTPBasedProtocolUsed now uses the new predicates instead of raw slice length checks:

• template.HasHTTPRequest() || template.HasHeadlessRequest() for top-level templates.
• Workflows still use TemplateType comparisons, which keeps the check inexpensive.

This is a straightforward readability win with no behaviour change.

Also applies to: 858-877

---

573-587: Workflow-aware validation changes are reasonable but do narrow coverage; verify this matches intended -validate behavior

In areWorkflowOrTemplatesValid (lines 573-587, 589-603, 606-617):

• The skip condition !isWorkflow && template.HasWorkflows() prevents templates with workflows from being validated when passed as regular templates. This is sensible only if templates containing workflows never appear in the non-workflow set during normal operation.
• areWorkflowTemplatesValid now recursively validates subtemplates via catalog lookups, which is reasonable.
• isParsingError excluding ErrCreateTemplateExecutor means templates that parse but cannot produce an executor now pass validation, narrowing what counts as a validation failure.

These changes reduce the validation scope compared to prior behavior. Confirm that:

1. Templates with .Workflows fields are only ever loaded into finalWorkflows, never into finalTemplates.
2. Executor-creation failures should not cause -validate to fail for templates in the current environment.

If templates can legitimately appear in both sets (e.g., a file containing both standard and workflow templates), this skip logic creates a coverage gap.

pkg/templates/templates.go (1)

413-503: ImportFileRefs flow/multiprotocol handling via IsFlowTemplate and HasRequest is clearer

Switching flow detection to template.IsFlowTemplate() and multiprotocol detection to HasRequest(template.RequestsQueue) makes the control flow much easier to follow:

• Flow templates: if Flow points to a .js file, it’s loaded via LoadHelperFile and options.Flow is set from the inlined content.
• Multiprotocol templates (non-flow) use RequestsQueue to resolve code/javascript file references consistently with single-protocol handling.

This keeps flow and multiprotocol concerns mutually exclusive and reuses the new helper predicates cleanly.

pkg/templates/compile.go (1)

323-352: Using HasXRequest in compileProtocolRequests keeps semantics while centralizing checks

Replacing direct length checks with HasDNSRequest/HasFileRequest/HasNetworkRequest/HasHTTPRequest/HasHeadlessRequest/HasSSLRequest/HasWebsocketRequest/HasWHOISRequest/HasCodeRequest/HasJavascriptRequest keeps the selection logic identical while reusing the new predicate helpers. The additional gating on options.Options.Headless and EnableCodeTemplates remains unchanged.