diff --git a/website/assets/robots.txt b/website/assets/robots.txt new file mode 100644 index 0000000000..eb892d30c8 --- /dev/null +++ b/website/assets/robots.txt @@ -0,0 +1,64 @@ +# As a condition of accessing this website, you agree to abide by the following +# content signals: + +# (a) If a Content-Signal = yes, you may collect content for the corresponding +# use. +# (b) If a Content-Signal = no, you may not collect content for the +# corresponding use. +# (c) If the website operator does not include a Content-Signal for a +# corresponding use, the website operator neither grants nor restricts +# permission via Content-Signal with respect to the corresponding use. + +# The content signals and their meanings are: + +# search: building a search index and providing search results (e.g., returning +# hyperlinks and short excerpts from your website's contents). Search does not +# include providing AI-generated search summaries. +# ai-input: inputting content into one or more AI models (e.g., retrieval +# augmented generation, grounding, or other real-time taking of content for +# generative AI search answers). +# ai-train: training or fine-tuning AI models. + +# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF +# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT +# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. + +User-agent: * +Content-Signal: search=yes,ai-train=no +Allow: / + +User-agent: Amazonbot +Disallow: / + +User-agent: Applebot-Extended +Disallow: / + +User-agent: Bytespider +Disallow: / + +User-agent: CCBot +Disallow: / + +User-agent: ClaudeBot +Disallow: / + +User-agent: Google-Extended +Disallow: / + +User-agent: GPTBot +Disallow: / + +User-agent: meta-externalagent +Disallow: / + +# ALLOW AI SEARCH & RETRIEVAL BOTS (sitewide) +User-agent: OAI-SearchBot +Allow: / + +User-agent: ChatGPT-User +Allow: / + +User-agent: PerplexityBot +Allow: / + +Sitemap: https://fleetdm.com/sitemap.xml