Feat/dynamic sitemap robots setup

[itsNintu]

Replace static sitemap and llms.txt generation with dynamic Next.js App Router implementation for both web client and docs applications. This implementation automatically discovers pages and
generates sitemaps +llms.txt +llms-full.txt + robots.txt files using Next.js built-in metadata routes.

Key Changes:

• Implement dynamic sitemap.ts and robots.ts files using Next.js App Router conventions
• Remove next-sitemap dependency from docs application
• Add automatic page discovery with configurable exclusion patterns
• Ensure proper SEO optimization with appropriate priorities and change frequencies
• Maintain consistency between robots.txt disallow rules and sitemap exclusions

Related Issues

Type of Change

[ ] Bug fix
[✓] New feature
[ ] Documentation update
[ ] Release
[✓] Refactor
[ ] Other (please describe):

Testing

Manual Testing Steps:

1. Web Client (onlook.com):
   • Visit /robots.txt - should show proper disallow rules and sitemap reference
   • Visit /sitemap.xml - should show all public pages with correct priorities
   • Verify excluded routes (auth, API, user-specific) are not in sitemap
2. Docs (docs.onlook.com):
   • Visit /robots.txt - should reference sitemap correctly
   • Visit /sitemap.xml - should show docs homepage
   • Verify old next-sitemap functionality is replaced
3. Build Testing:
   • Run bun install and bun build for both applications
   • Confirm no next-sitemap related errors in docs build

Screenshots (if applicable)

Additional Notes

• Breaking Change: Removes next-sitemap dependency - docs application no longer needs postbuild script
• SEO Optimized: Homepage gets priority 1.0, marketing pages 0.9, auth pages 0.6
• Automatic: New pages are automatically included in sitemap without manual configuration
• Secure: Private routes (user dashboards, API endpoints) are automatically excluded
• Standards Compliant: Uses official Next.js metadata route conventions for better caching and performance
anthropic/claude-4-sonnet-20250522 (07:39 AM)

---

Important

Replaces static sitemap generation with dynamic Next.js App Router implementation, removing next-sitemap dependency and adding automatic page discovery with exclusion patterns.

• Behavior:
  • Replace static sitemap generation with dynamic Next.js App Router implementation in sitemap.ts and robots.ts.
  • Remove next-sitemap dependency from docs application.
  • Add automatic page discovery with exclusion patterns.
  • Ensure SEO optimization with priorities and change frequencies.
  • Maintain consistency between robots.txt disallow rules and sitemap exclusions.
• Files:
  • sitemap.ts, robots.ts, and sitemap-utils.ts in apps/web/client/src/app for dynamic sitemap and robots.txt generation.
  • llms-full.txt/route.ts and llms.txt/route.ts in apps/web/client/src/app and docs/src/app for documentation routes.
  • Remove next-sitemap.config.js from docs.
• Audit Report:
  • Highlights architecture, URL generation logic, file system access, and performance implications.
  • Recommends environment-aware URL generation, response caching, and content consistency.

^{This description was created by for 9e8b196. You can customize this summary. It will automatically update as commits are pushed.}

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 3, 2025 6:34am
web	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 3, 2025 6:34am

[supabase]

This pull request has been ignored for the connected project wowaemfasoptxrdjhilu because there are no changes detected in apps/backend/supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.