robots-txt flags the Agentmap directive that ard-schema reads #17480

Submission Type

Feedback / Bug Report on Existing Audit

Audit Name / Affected Audit

robots-txt (SEO) and ard-schema (Agentic Browsing)

Description & Rationale

The ARD spec (v0.91, §5.1) lets a site point agents to its manifest with a robots.txt line such as Agentmap: https://example.com/.well-known/ard.json, and the ard-schema gatherer reads that line first (getRobotsTxtAgentmap in core/gather/gatherers/agentic/ard.js).

The robots-txt audit fails the same file: "robots.txt is not valid", with "Unknown directive" on the Agentmap line, because agentmap is not in DIRECTIVE_SAFELIST (core/audits/seo/robots-txt.js). So a site that follows the spec fails the robots-txt audit, which weighs about 8% of the SEO score (Lighthouse 13.6.0).

Google ignores the line, so it harms nothing: "Rules other than allow, disallow, and user-agent are ignored by the robots.txt parser." (https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)

Target Agents

No response

Proposed Implementation / Suggested Improvement

Add agentmap to DIRECTIVE_SAFELIST, as was done for content-signal (#16767) and license (#17160).

Example URL

No response