Skip to content

Replace local ai-agent-detection with @vercel/agent-readability #354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
molebox wants to merge 5 commits into
base: main
Choose a base branch
from
+38 −175
Open

Replace local ai-agent-detection with @vercel/agent-readability #354

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
01497db
Add @vercel/agent-readability dependency
molebox Apr 1, 2026
0087043
Replace local isAIAgent with @vercel/agent-readability
molebox Apr 1, 2026
9e09bdc
Delete local ai-agent-detection.ts (replaced by package)
molebox Apr 1, 2026
73c38b6
Fix broken DetectionMethod import, add Vary header and discovery options
molebox Apr 7, 2026
0a14f7a
Update pnpm-lock.yaml
molebox Apr 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 0 additions & 168 deletions apps/docs/lib/ai-agent-detection.ts
View file
Open in desktop

This file was deleted.

3 changes: 2 additions & 1 deletion apps/docs/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
"@orama/tokenizers": "^3.1.16",
"@streamdown/cjk": "^1.0.1",
"@streamdown/code": "^1.0.1",
"@vercel/agent-readability": "^0.2.1",
"@vercel/analytics": "^1.6.1",
"@vercel/speed-insights": "^1.3.1",
"@vercel/toolbar": "0.1.36",
Expand Down Expand Up @@ -60,4 +61,4 @@
"typescript": "^5.9.3"
},
"packageManager": "pnpm@10.19.0"
}
}
10 changes: 6 additions & 4 deletions apps/docs/proxy.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { generateNotFoundMarkdown, isAIAgent } from "@vercel/agent-readability";
import { precompute } from "flags/next";
import { createI18nMiddleware } from "fumadocs-core/i18n/middleware";
import { isMarkdownPreferred, rewritePath } from "fumadocs-core/negotiation";
Expand All @@ -8,7 +9,6 @@ import {
} from "next/server";
import { rootFlags } from "@/flags";
import { i18n } from "@/lib/geistdocs/i18n";
import { isAIAgent } from "@/lib/ai-agent-detection";
import { trackMdRequest } from "@/lib/geistdocs/md-tracking";

const { rewrite: rewriteLLM } = rewritePath(
Expand Down Expand Up @@ -65,7 +65,6 @@ const proxy = async (request: NextRequest, context: NextFetchEvent) => {
}

// AI agent detection — rewrite docs pages to markdown for agents
// so they always get structured content without needing .md URLs or Accept headers
if (
(pathname === "/docs" || pathname.startsWith("/docs/")) &&
!pathname.includes("/llms.mdx/")
Expand All @@ -90,6 +89,10 @@ const proxy = async (request: NextRequest, context: NextFetchEvent) => {
);
return NextResponse.rewrite(new URL(result, request.nextUrl));
}
// Agent requested a non-existent docs URL — return helpful markdown
return new NextResponse(generateNotFoundMarkdown(pathname), {
Copy link
Copy Markdown
Contributor

@vercel vercel Bot Apr 1, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI agent not-found response returns HTTP 200 instead of 404, causing agents to incorrectly believe the page exists.

Fix on Vercel

Copy link
Copy Markdown
Collaborator

@dferber90 dferber90 Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@molebox should these be 404s?

Copy link
Copy Markdown
Contributor

@vercel vercel Bot Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes — the comment on line 92 says "Agent requested a non-existent docs URL" and the function is called generateNotFoundMarkdown, so this is clearly a 404 scenario. Without an explicit status: 404, NextResponse defaults to 200, which would mislead AI agents that check status codes to determine if a resource exists.

The fix is adding status: 404 to the response options (see the suggestion above).

headers: { "Content-Type": "text/markdown; charset=utf-8" },
});
}
}

Expand All @@ -115,10 +118,9 @@ const proxy = async (request: NextRequest, context: NextFetchEvent) => {
};

export const config = {
// Matcher ignoring `/_next/`, `/api/`, static assets, favicon, sitemap, robots, etc.
matcher: [
"/((?!api|_next/static|_next/image|favicon.ico|sitemap.xml|robots.txt|\\.well-known/vercel/flags).*)",
],
};

export default proxy;
export default proxy;
Loading