A web application that automatically generates llms.txt files from any website's sitemap. Built with Next.js, TypeScript, and Tailwind CSS.
LLMS Generator is a free, no-code tool that helps website owners create llms.txt files for LLM (Large Language Model) SEO compliance. Simply provide your website's sitemap URL, and the tool will:
- Crawl your website pages
- Extract titles and descriptions
- Categorize content by sections
- Generate a properly formatted
llms.txtfile - Provide real-time progress tracking
- Automatic Content Extraction: Fetches and parses HTML content from your website pages
- Smart Categorization: Automatically categorizes pages into logical sections (Home, About, Services, Blog, etc.)
- Real-time Progress: Live progress tracking with detailed status updates
- Configurable Options: Customize concurrency, priority thresholds, and content filtering
- Modern UI: Clean, responsive interface built with Tailwind CSS and ShadcnUI
- Free & Open Source: No registration required, completely free to use
- Frontend: Next.js 15, React 19, TypeScript
- Styling: Tailwind CSS 4, ShadcnUI components
- Content Processing: Cheerio (HTML parsing), Sitemapper (sitemap fetching)
- Deployment: Ready for Vercel, Netlify, or any Next.js hosting platform
- Node.js 18+
- npm or yarn
- Clone the repository:
git clone https://github.com/yourusername/llms-generator.git
cd llms-generator- Install dependencies:
npm install- Run the development server:
npm run dev- Open http://localhost:3000 in your browser.
npm run build
npm start-
Enter Sitemap URL: Provide your website's sitemap URL (e.g.,
https://example.com/sitemap.xml) -
Configure Options (Optional):
- Concurrency: Number of simultaneous requests (default: 8)
- Priority Threshold: Minimum priority for pages to include (default: 0.3)
- Include/Exclude Paths: Filter specific URL patterns
- Custom Title/Description: Override auto-generated site metadata
-
Generate: Click "Generate LLMS.txt" and watch the real-time progress
-
Download: Once complete, download your generated
llms.txtfile
const DEFAULT_CONFIG = {
concurrency: 8, // Simultaneous requests
priorityThreshold: 0.3, // Minimum page priority
requestTimeout: 10000, // Request timeout in ms
userAgent: "Mozilla/5.0 (compatible; LLMS-txt-Generator/1.0)",
};The tool automatically categorizes pages into these sections:
- Home: Main landing pages
- About: Company information, team, mission
- Services: Products, pricing, solutions
- Guides: Tutorials, documentation, how-to content
- Blog: Articles, news, updates
- Contact: Support, help, contact forms
- Legal: Privacy policy, terms of service
- Resources: Downloads, tools, assets
llms-generator/
├── app/ # Next.js app directory
│ ├── api/ # API routes
│ │ └── parse/ # Sitemap processing endpoint
│ ├── page.tsx # Main page component
│ └── layout.tsx # Root layout
├── components/ # React components
│ ├── common/ # Shared components
│ ├── features/ # Feature-specific components
│ ├── forms/ # Form components
│ ├── layout/ # Layout components
│ └── ui/ # UI primitives (Shadcn UI)
├── hooks/ # Custom React hooks
├── lib/ # Utility functions and types
│ ├── constants.ts # Configuration constants
│ ├── sitemap-processor.ts # Core processing logic
│ ├── types.ts # TypeScript type definitions
│ └── utils.ts # Utility functions
└── public/ # Static assets
The application exposes a single API endpoint for processing sitemaps:
Request Body:
{
sitemapUrl: string;
options?: {
concurrency?: number;
includePaths?: string[];
excludePaths?: string[];
customTitle?: string;
customDescription?: string;
priorityThreshold?: number;
};
}Response: Server-sent events stream with progress updates and final result.
Contributions are welcome! Please feel free to submit a Pull Request.
- Fork the repository
- Create a feature branch:
git checkout -b feature/amazing-feature - Commit your changes:
git commit -m 'Add amazing feature' - Push to the branch:
git push origin feature/amazing-feature - Open a Pull Request
This project is licensed under the MIT License - see the LICENSE file for details.
- Built with Next.js
- UI components from ShadcnUI
- Styling with Tailwind CSS
- Icons from Lucide React
If you have any questions or need help, please:
- Check the Issues page
- Create a new issue if your problem isn't already addressed
- Join our community discussions
Made with ❤️ for the LLM SEO community