The app is pre-launch and shared by link with a handful of people at Prime
Intellect. It should not be accumulating a search footprint yet.
Three layers, because each covers a gap the others leave:
- robots.txt asks well-behaved crawlers not to fetch at all.
- The <meta name="robots"> tag covers the HTML document for anything that
fetched anyway.
- X-Robots-Tag covers everything that is NOT the HTML document — og.png,
the manifest, the built assets — which the meta tag cannot reach.
noarchive and nosnippet are there so a cache or an excerpt cannot outlive
the page once this is reversed.
Deliberately NOT stripped: the og:/twitter: tags. Link unfurlers are not
crawlers — they fetch on behalf of the person pasting the link, and a
rendered card is exactly what we want when this is shared.
The real gate remains authentication: / returns the sign-in screen and every
/api/ route returns 401. This only stops the app being indexed.
To go public: delete robots.txt, drop the meta tag, drop the header.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
