Add llms.txt

[gaearon]

Written by Claude. Pretty minimal setup.

Test plan

[github-actions]

Size changes

Details

📦 Next.js Bundle Analysis for react-dev

This analysis was generated by the Next.js Bundle Analysis action. 🤖

⚠️ Global Bundle Size Increased

Page	Size (compressed)
global	115.15 KB (🟡 +4.6 KB)

Details

The global bundle is the javascript bundle that loads alongside every page. It is in its own category because its impact is much higher - an increase to its size means that every page on your website loads slower, and a decrease means every page loads faster.

Any third party scripts you have added directly to your app using the <script> tag are not accounted for in this analysis

If you want further insight into what is behind the changes, give @next/bundle-analyzer a try!

New Page Added

The following page was added to the bundle from the code in this PR:

Page	Size (compressed)	First Load
/llms.txt	253 B	115.4 KB

[gaearon]

apparently i don't have access to see why deploy failed so i can't fix it, worked fine locally

[rickhanlonii]

@gaearon I'll fix your permissions.

The failure was because of this: #8268

Rebase to main and it should deploy

[rickhanlonii]

Nice, i tweeted about this and there were other PRs opened.

This is like @artimath's approach, but this is a bit cleaner IMO

• #8130
• #8133

Let's add @artimath as co-author on merge.

[rickhanlonii]

Looks good in the preview:

[gaearon]

yea sorry, i haven't searched, it just annoyed me today

probably need to add more granular grouping, useful for both Learn and Reference

[rickhanlonii]

Yeah, since it uses sidebar*.json it could just use the grouping there - let me add that now.

[leerob]

Two other things you might consider:

1. Adding a "sitemap"-like footer for the agent at the bottom of files

---

## Sitemap

[Overview of all docs pages](/llms.txt)

Example: https://cursor.com/docs/context/skills.md

2. Allow humans & agents to cURL for markdown

curl -H "Accept: text/markdown" https://cursor.com/docs/context/skills

Can implement these if you want!

[rickhanlonii]

Sick @leerob!

I added the section grouping and sitemap footer here: #8270

I could use help with the Accept: text/markdown tho, I don't know how to do that with nextjs offhand.

[leerob]

Oh nice, @icyJoseph is on it! #8272

[rickhanlonii]

Merged, but that doesn't work for some reason #8272 (comment)

[icyJoseph]

I fixed it here, #8276 - needed to rewrite w/ beforeFiles