From 6cf80747cc2de18c9b8810666ba9fd25cf97d368 Mon Sep 17 00:00:00 2001 From: Kartios Date: Thu, 13 Aug 2026 12:45:18 -0700 Subject: [PATCH] Keep PIG out of search results until it is meant to be found MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The app is pre-launch and shared by link with a handful of people at Prime Intellect. It should not be accumulating a search footprint yet. Three layers, because each covers a gap the others leave: - robots.txt asks well-behaved crawlers not to fetch at all. - The tag covers the HTML document for anything that fetched anyway. - X-Robots-Tag covers everything that is NOT the HTML document — og.png, the manifest, the built assets — which the meta tag cannot reach. noarchive and nosnippet are there so a cache or an excerpt cannot outlive the page once this is reversed. Deliberately NOT stripped: the og:/twitter: tags. Link unfurlers are not crawlers — they fetch on behalf of the person pasting the link, and a rendered card is exactly what we want when this is shared. The real gate remains authentication: / returns the sign-in screen and every /api/ route returns 401. This only stops the app being indexed. To go public: delete robots.txt, drop the meta tag, drop the header. Co-Authored-By: Claude Opus 5 (1M context) --- apps/web/index.html | 13 +++++++++++++ apps/web/public/robots.txt | 17 +++++++++++++++++ deploy/Caddyfile.example | 5 +++++ 3 files changed, 35 insertions(+) create mode 100644 apps/web/public/robots.txt diff --git a/apps/web/index.html b/apps/web/index.html index ee57059..76272f1 100644 --- a/apps/web/index.html +++ b/apps/web/index.html @@ -15,6 +15,19 @@ + + + diff --git a/apps/web/public/robots.txt b/apps/web/public/robots.txt new file mode 100644 index 0000000..2dd1d1c --- /dev/null +++ b/apps/web/public/robots.txt @@ -0,0 +1,17 @@ +# PIG is not public yet. Nothing here should be indexed or crawled. +# +# This is a request, not enforcement — well-behaved crawlers honour it, and +# hostile ones do not. The real gate is authentication: every route below / +# returns the sign-in screen to an unauthenticated visitor and every /api/ +# route returns 401. This file exists so the app does not accumulate a search +# footprint before it is meant to have one. +# +# Backed by an `X-Robots-Tag: noindex, nofollow` response header in the Caddy +# config and a tag in index.html. The header is the one +# that matters most: it also covers og.png, the manifest and anything else +# served that is not HTML. +# +# To go public: delete this file, remove the meta tag, and drop the header. + +User-agent: * +Disallow: / diff --git a/deploy/Caddyfile.example b/deploy/Caddyfile.example index e3557f1..7b84732 100644 --- a/deploy/Caddyfile.example +++ b/deploy/Caddyfile.example @@ -20,6 +20,11 @@ primeintellectgrowth.com, www.primeintellectgrowth.com { # IMPORTANT: editing that script changes its hash and CSP will silently # block it. The browser console says exactly which hash it wants. header { + # Not public yet. The header covers everything served — og.png, the + # manifest, the built assets — where the tag in index.html only + # covers the HTML document. robots.txt asks crawlers not to fetch; + # this tells the ones that fetch anyway not to index. + X-Robots-Tag "noindex, nofollow, noarchive, nosnippet" Strict-Transport-Security "max-age=31536000; includeSubDomains" X-Content-Type-Options "nosniff" X-Frame-Options "DENY"