The app is pre-launch and shared by link with a handful of people at Prime
Intellect. It should not be accumulating a search footprint yet.
Three layers, because each covers a gap the others leave:
- robots.txt asks well-behaved crawlers not to fetch at all.
- The <meta name="robots"> tag covers the HTML document for anything that
fetched anyway.
- X-Robots-Tag covers everything that is NOT the HTML document — og.png,
the manifest, the built assets — which the meta tag cannot reach.
noarchive and nosnippet are there so a cache or an excerpt cannot outlive
the page once this is reversed.
Deliberately NOT stripped: the og:/twitter: tags. Link unfurlers are not
crawlers — they fetch on behalf of the person pasting the link, and a
rendered card is exactly what we want when this is shared.
The real gate remains authentication: / returns the sign-in screen and every
/api/ route returns 401. This only stops the app being indexed.
To go public: delete robots.txt, drop the meta tag, drop the header.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -20,6 +20,11 @@ primeintellectgrowth.com, www.primeintellectgrowth.com {
|
||||
# IMPORTANT: editing that script changes its hash and CSP will silently
|
||||
# block it. The browser console says exactly which hash it wants.
|
||||
header {
|
||||
# Not public yet. The header covers everything served — og.png, the
|
||||
# manifest, the built assets — where the <meta> tag in index.html only
|
||||
# covers the HTML document. robots.txt asks crawlers not to fetch;
|
||||
# this tells the ones that fetch anyway not to index.
|
||||
X-Robots-Tag "noindex, nofollow, noarchive, nosnippet"
|
||||
Strict-Transport-Security "max-age=31536000; includeSubDomains"
|
||||
X-Content-Type-Options "nosniff"
|
||||
X-Frame-Options "DENY"
|
||||
|
||||
Reference in New Issue
Block a user