|
| 1 | +--- |
| 2 | +description: Generate or update llms.txt file to help LLMs understand and navigate your site |
| 3 | +--- |
| 4 | + |
| 5 | +# Generate llms.txt |
| 6 | + |
| 7 | +Creates or updates `llms.txt` - a standardized file that helps Large Language Models understand and navigate your website or documentation at inference time. |
| 8 | + |
| 9 | +<philosophy> |
| 10 | +llms.txt provides LLM-friendly navigation and context for websites and documentation. It should be: |
| 11 | +- Concise - LLMs have limited context windows |
| 12 | +- Navigational - Links to key documentation rather than duplicating it |
| 13 | +- Structured - Uses markdown format for easy LLM parsing |
| 14 | +- Focused - Highlights most important resources first |
| 15 | + |
| 16 | +Think: "What does an LLM need to understand and effectively use this site?" |
| 17 | +</philosophy> |
| 18 | + |
| 19 | +<workflow> |
| 20 | +<determine-location> |
| 21 | +Detect the appropriate location for llms.txt based on project type: |
| 22 | + |
| 23 | +1. Check for `public/` directory (Next.js, React, Vue, many static site generators) |
| 24 | +2. Check for `static/` directory (Django, Flask, Hugo) |
| 25 | +3. Check for `docs/` directory if this appears to be a documentation site |
| 26 | +4. Default to repository root if none found |
| 27 | + |
| 28 | +The file should be placed where it will be accessible at the URL path `/llms.txt` when the site is deployed. |
| 29 | +</determine-location> |
| 30 | + |
| 31 | +<analyze-project> |
| 32 | +Understand the project structure to generate relevant content: |
| 33 | + |
| 34 | +1. Detect project type: |
| 35 | + - Check `package.json` for web frameworks (Next.js, React, Vue, etc.) |
| 36 | + - Check `pyproject.toml`, `requirements.txt` for Python frameworks (Django, Flask, FastAPI) |
| 37 | + - Check for static site generators (Hugo, Jekyll, Gatsby, etc.) |
| 38 | + - Check for documentation tools (MkDocs, Docusaurus, VitePress, etc.) |
| 39 | + |
| 40 | +2. Identify key documentation: |
| 41 | + - Look for README.md |
| 42 | + - Look for docs/ or documentation/ directory |
| 43 | + - Look for CONTRIBUTING.md, ARCHITECTURE.md, API.md |
| 44 | + - Check .cursor/rules/ for coding standards |
| 45 | + - Check .claude/agents/ for specialized agents |
| 46 | + - Check for API documentation (OpenAPI/Swagger files) |
| 47 | + |
| 48 | +3. Identify important links: |
| 49 | + - GitHub repository URL (from package.json, git remote) |
| 50 | + - Documentation site URL (from package.json, README) |
| 51 | + - API reference locations |
| 52 | + - Getting started guides |
| 53 | +</analyze-project> |
| 54 | + |
| 55 | +<generate-structure> |
| 56 | +Create llms.txt following the standard format: |
| 57 | + |
| 58 | +```markdown |
| 59 | +# Project Name |
| 60 | + |
| 61 | +> Brief 1-2 sentence description of what this project/site is and what it helps users accomplish. |
| 62 | + |
| 63 | +Additional context paragraph (optional): More detailed information about the project's purpose, key features, or important background that helps LLMs understand how to use the site effectively. |
| 64 | + |
| 65 | +## Documentation |
| 66 | + |
| 67 | +- [Getting Started](url): Brief description of what this covers |
| 68 | +- [Core Concepts](url): Brief description |
| 69 | +- [API Reference](url): Brief description |
| 70 | + |
| 71 | +## Guides |
| 72 | + |
| 73 | +- [Guide Title](url): Brief description |
| 74 | +- [Another Guide](url): Brief description |
| 75 | + |
| 76 | +## Reference |
| 77 | + |
| 78 | +- [Configuration](url): Brief description |
| 79 | +- [CLI Commands](url): Brief description |
| 80 | +- [Architecture](url): Brief description |
| 81 | + |
| 82 | +## Optional |
| 83 | + |
| 84 | +- [Less Critical Resource](url): Brief description |
| 85 | +- [Advanced Topics](url): Brief description |
| 86 | +``` |
| 87 | + |
| 88 | +Structure guidelines: |
| 89 | +- **H1 title** (required): Project or site name |
| 90 | +- **Blockquote** (recommended): Short summary with key information for understanding the rest of the file |
| 91 | +- **Additional context** (optional): Detailed background if needed |
| 92 | +- **Sections** (H2 headers): Organize documentation by category |
| 93 | + - Common categories: Documentation, Guides, Reference, API, Examples, Contributing |
| 94 | + - Use "Optional" section for less critical resources |
| 95 | +- **Links**: Markdown format with optional brief descriptions |
| 96 | + - Use relative paths for internal documentation |
| 97 | + - Use absolute URLs for external resources |
| 98 | + - Keep descriptions concise |
| 99 | + |
| 100 | +Priority order: |
| 101 | +1. Getting started / quickstart guides |
| 102 | +2. Core concepts and tutorials |
| 103 | +3. API reference and technical documentation |
| 104 | +4. Advanced topics and edge cases |
| 105 | +5. Contributing and community resources |
| 106 | +</generate-structure> |
| 107 | + |
| 108 | +<create-llms-full-txt> |
| 109 | +Optionally create llms-full.txt alongside llms.txt: |
| 110 | + |
| 111 | +- `llms.txt` - Lightweight navigation file (links to docs) |
| 112 | +- `llms-full.txt` - Comprehensive file with full documentation content |
| 113 | + |
| 114 | +For most projects, start with just `llms.txt`. Only create `llms-full.txt` if: |
| 115 | +- Project has extensive documentation that would benefit from a single-file view |
| 116 | +- Documentation is already in markdown format |
| 117 | +- Total size would be reasonable (< 100KB) |
| 118 | + |
| 119 | +Skip llms-full.txt creation for now unless user specifically requests it. |
| 120 | +</create-llms-full-txt> |
| 121 | + |
| 122 | +<optimize-for-llms> |
| 123 | +Review and optimize the generated content: |
| 124 | + |
| 125 | +1. **Prioritize ruthlessly**: Most important links first |
| 126 | +2. **Be concise**: Brief descriptions, not full explanations |
| 127 | +3. **Use relative paths**: For internal documentation (easier to maintain) |
| 128 | +4. **Organize logically**: Group related topics together |
| 129 | +5. **Mark optional content**: Use "Optional" section for nice-to-have resources |
| 130 | +6. **Keep it current**: Link to actively maintained documentation |
| 131 | +7. **Test URLs**: Ensure all links will be valid when deployed |
| 132 | + |
| 133 | +Target: Under 2KB for most projects (focused navigation, not content duplication) |
| 134 | +</optimize-for-llms> |
| 135 | + |
| 136 | +<report> |
| 137 | +Show the user: |
| 138 | +1. Where the file will be created (with explanation of why) |
| 139 | +2. The generated content |
| 140 | +3. File size |
| 141 | +4. Suggestions for what to add if content seems sparse |
| 142 | +5. Next steps (how to verify it works, how to update it) |
| 143 | +</report> |
| 144 | +</workflow> |
| 145 | + |
| 146 | +<update-mode> |
| 147 | +When `llms.txt` already exists: |
| 148 | + |
| 149 | +1. Read existing file to understand current content |
| 150 | +2. Analyze project for changes: |
| 151 | + - New documentation files added |
| 152 | + - Changed project structure |
| 153 | + - Updated README or documentation |
| 154 | + - New major features or guides |
| 155 | +3. Suggest additions or updates with rationale |
| 156 | +4. Show diff of proposed changes |
| 157 | +5. Let user review before updating |
| 158 | + |
| 159 | +Preserve existing structure and organization unless there's a good reason to change it. |
| 160 | +</update-mode> |
| 161 | + |
| 162 | +<key-principles> |
| 163 | +Navigation over duplication: Link to documentation, don't duplicate it. |
| 164 | + |
| 165 | +LLM-friendly format: Markdown is the most widely understood format for LLMs. |
| 166 | + |
| 167 | +Prioritize ruthlessly: LLMs have limited context - put most important content first. |
| 168 | + |
| 169 | +Keep it current: Outdated links are worse than missing links. |
| 170 | + |
| 171 | +Test the result: After generating, consider "Would an LLM understand how to navigate this site?" |
| 172 | +</key-principles> |
| 173 | + |
| 174 | +<location-detection-examples> |
| 175 | +Example directory structures and where to place llms.txt: |
| 176 | + |
| 177 | +Next.js project: |
| 178 | +``` |
| 179 | +project/ |
| 180 | +├── public/ ← Place llms.txt here |
| 181 | +├── src/ |
| 182 | +└── package.json |
| 183 | +``` |
| 184 | + |
| 185 | +Django project: |
| 186 | +``` |
| 187 | +project/ |
| 188 | +├── static/ ← Place llms.txt here |
| 189 | +├── myapp/ |
| 190 | +└── manage.py |
| 191 | +``` |
| 192 | + |
| 193 | +Documentation site: |
| 194 | +``` |
| 195 | +project/ |
| 196 | +├── docs/ ← Consider placing here if this is the deployed location |
| 197 | +├── README.md |
| 198 | +└── mkdocs.yml |
| 199 | +``` |
| 200 | + |
| 201 | +Plain repository (no public/static): |
| 202 | +``` |
| 203 | +project/ |
| 204 | +├── llms.txt ← Place in root |
| 205 | +├── README.md |
| 206 | +└── src/ |
| 207 | +``` |
| 208 | +</location-detection-examples> |
| 209 | + |
| 210 | +<common-sections> |
| 211 | +Recommended section organization based on project type: |
| 212 | + |
| 213 | +**API/Library Project:** |
| 214 | +- Documentation (getting started, core concepts) |
| 215 | +- API Reference |
| 216 | +- Examples |
| 217 | +- Guides |
| 218 | +- Optional (advanced topics, contributing) |
| 219 | + |
| 220 | +**Web Application:** |
| 221 | +- Documentation (overview, quickstart) |
| 222 | +- User Guides |
| 223 | +- API Documentation (if applicable) |
| 224 | +- Architecture |
| 225 | +- Optional (deployment, troubleshooting) |
| 226 | + |
| 227 | +**Documentation Site:** |
| 228 | +- Getting Started |
| 229 | +- Tutorials |
| 230 | +- Reference |
| 231 | +- Guides |
| 232 | +- Optional (FAQ, community) |
| 233 | + |
| 234 | +**Developer Tool/CLI:** |
| 235 | +- Documentation (installation, quickstart) |
| 236 | +- Commands Reference |
| 237 | +- Configuration |
| 238 | +- Examples |
| 239 | +- Optional (plugins, advanced usage) |
| 240 | +</common-sections> |
| 241 | + |
| 242 | +<final-checklist> |
| 243 | +- File location is appropriate for project type |
| 244 | +- H1 title reflects project name |
| 245 | +- Blockquote provides clear project summary |
| 246 | +- Sections are logically organized |
| 247 | +- Most important links come first |
| 248 | +- "Optional" section for less critical resources |
| 249 | +- Link descriptions are concise and helpful |
| 250 | +- All URLs will be valid when site is deployed |
| 251 | +- File is under 2KB (focused navigation) |
| 252 | +- Format follows llms.txt specification from llmstxt.org |
| 253 | +</final-checklist> |
0 commit comments