# RateMate robots.txt
#
# Everything public is open to every crawler, including the ones that train on
# what they read. This used to be split — GPTBot/ClaudeBot/CCBot/Google-Extended
# were blocked from training while answer crawlers were welcomed — on the theory
# that training ingests our content into model weights with no link back.
#
# That theory misprices what's being protected. The moat is the authorized ConEd
# data connection, the rate engine, and actually executing the switch; it is not
# the prose. Nobody reads "the median ESCO overpay is about $21 a month" in a
# chatbot and thereby solves their problem — they still need someone with access
# to their account. Our guides exist to be found and believed, so a model that
# has absorbed them is doing our marketing, not stealing our product.
#
# The cost of the old policy was concrete: assistants asked to recommend an
# independent ConEd rate advisor named competitors from memory and reached us
# only if they happened to run a live search. Being retrievable is not the same
# as being known. Note this is a one-way door — weights don't forget — and slow,
# since it only lands with a future training run.
#
# Private/personal pages (/share, /results, /progress) are kept OUT of search
# via X-Robots-Tag: noindex (see next.config.ts), NOT via Disallow here — a
# Disallowed page can't be crawled, so its noindex would never be read.
#
# /results and /progress sit behind auth, so the header is a second layer there.
# /share does NOT: the token IS the credential, and anyone holding the URL gets
# the report anonymously (GET /api/payments/share/:token). For share links the
# noindex header is the ONLY thing keeping them out of search — and it only
# binds crawlers that read it. Headless fleets that ignore robots directives
# entirely do fetch these URLs; stopping those needs token expiry or
# rate-limiting on the API, not a line in this file.
#
# One group, no per-bot exceptions. Content-Signal is scoped to the group it sits
# in, so a crawler with its own named group would not see these directives at all
# — naming bots here would silently exempt them from the policy below.
#   ai-train=yes  — training on this content is permitted
#   search=yes    — index it and link to it
#   ai-input=yes  — fetch it live to answer a user's question and cite us
#                   (this is the whole point: we want to be the answer when
#                   someone asks a chatbot why their ConEd bill is so high)
User-agent: *
Content-Signal: ai-train=yes, search=yes, ai-input=yes
Allow: /

Sitemap: https://ratemate.energy/sitemap.xml