karti
6cf80747cc
CI / verify (push) Successful in 3m11s
The app is pre-launch and shared by link with a handful of people at Prime
Intellect. It should not be accumulating a search footprint yet.
Three layers, because each covers a gap the others leave:
- robots.txt asks well-behaved crawlers not to fetch at all.
- The <meta name="robots"> tag covers the HTML document for anything that
fetched anyway.
- X-Robots-Tag covers everything that is NOT the HTML document — og.png,
the manifest, the built assets — which the meta tag cannot reach.
noarchive and nosnippet are there so a cache or an excerpt cannot outlive
the page once this is reversed.
Deliberately NOT stripped: the og:/twitter: tags. Link unfurlers are not
crawlers — they fetch on behalf of the person pasting the link, and a
rendered card is exactly what we want when this is shared.
The real gate remains authentication: / returns the sign-in screen and every
/api/ route returns 401. This only stops the app being indexed.
To go public: delete robots.txt, drop the meta tag, drop the header.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
18 lines
785 B
Plaintext
18 lines
785 B
Plaintext
# PIG is not public yet. Nothing here should be indexed or crawled.
|
|
#
|
|
# This is a request, not enforcement — well-behaved crawlers honour it, and
|
|
# hostile ones do not. The real gate is authentication: every route below /
|
|
# returns the sign-in screen to an unauthenticated visitor and every /api/
|
|
# route returns 401. This file exists so the app does not accumulate a search
|
|
# footprint before it is meant to have one.
|
|
#
|
|
# Backed by an `X-Robots-Tag: noindex, nofollow` response header in the Caddy
|
|
# config and a <meta name="robots"> tag in index.html. The header is the one
|
|
# that matters most: it also covers og.png, the manifest and anything else
|
|
# served that is not HTML.
|
|
#
|
|
# To go public: delete this file, remove the meta tag, and drop the header.
|
|
|
|
User-agent: *
|
|
Disallow: /
|