Keep PIG out of search results until it is meant to be found

The app is pre-launch and shared by link with a handful of people at Prime
Intellect. It should not be accumulating a search footprint yet.

Three layers, because each covers a gap the others leave:

  - robots.txt asks well-behaved crawlers not to fetch at all.
  - The <meta name="robots"> tag covers the HTML document for anything that
    fetched anyway.
  - X-Robots-Tag covers everything that is NOT the HTML document — og.png,
    the manifest, the built assets — which the meta tag cannot reach.

noarchive and nosnippet are there so a cache or an excerpt cannot outlive
the page once this is reversed.

Deliberately NOT stripped: the og:/twitter: tags. Link unfurlers are not
crawlers — they fetch on behalf of the person pasting the link, and a
rendered card is exactly what we want when this is shared.

The real gate remains authentication: / returns the sign-in screen and every
/api/ route returns 401. This only stops the app being indexed.

To go public: delete robots.txt, drop the meta tag, drop the header.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

apps/web/index.html

@@ -15,6 +15,19 @@
     <meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover" />
     <meta name="description" content="PIG — Prime Intellect Growth. An agent-native CRM for two-sided AI-compute companies." />
 
+    <!--
+      Not public yet. noindex keeps the app out of search results; noarchive
+      and nosnippet stop a cache or excerpt surviving after it is removed.
+      Mirrored by /robots.txt and by an X-Robots-Tag header in Caddy — see the
+      comment in robots.txt for why all three exist.
+
+      Note this deliberately does NOT strip the og:/twitter: tags below. Link
+      unfurlers are not crawlers: they fetch on behalf of the person pasting
+      the link, and a card is exactly what we want when this is shared with
+      Prime Intellect.
+    -->
+    <meta name="robots" content="noindex, nofollow, noarchive, nosnippet" />
+
     <!-- Matches the app chrome so Safari's toolbar blends rather than banding. -->
     <meta name="theme-color" content="#ffffff" media="(prefers-color-scheme: light)" />
     <meta name="theme-color" content="#09090b" media="(prefers-color-scheme: dark)" />

apps/web/public/robots.txt

@@ -0,0 +1,17 @@
+# PIG is not public yet. Nothing here should be indexed or crawled.
+#
+# This is a request, not enforcement — well-behaved crawlers honour it, and
+# hostile ones do not. The real gate is authentication: every route below /
+# returns the sign-in screen to an unauthenticated visitor and every /api/
+# route returns 401. This file exists so the app does not accumulate a search
+# footprint before it is meant to have one.
+#
+# Backed by an `X-Robots-Tag: noindex, nofollow` response header in the Caddy
+# config and a <meta name="robots"> tag in index.html. The header is the one
+# that matters most: it also covers og.png, the manifest and anything else
+# served that is not HTML.
+#
+# To go public: delete this file, remove the meta tag, and drop the header.
+
+User-agent: *
+Disallow: /

deploy/Caddyfile.example

@@ -20,6 +20,11 @@ primeintellectgrowth.com, www.primeintellectgrowth.com {
 	# IMPORTANT: editing that script changes its hash and CSP will silently
 	# block it. The browser console says exactly which hash it wants.
 	header {
+		# Not public yet. The header covers everything served — og.png, the
+		# manifest, the built assets — where the <meta> tag in index.html only
+		# covers the HTML document. robots.txt asks crawlers not to fetch;
+		# this tells the ones that fetch anyway not to index.
+		X-Robots-Tag "noindex, nofollow, noarchive, nosnippet"
 		Strict-Transport-Security "max-age=31536000; includeSubDomains"
 		X-Content-Type-Options "nosniff"
 		X-Frame-Options "DENY"