Obituary: Farewell to robots.txt (1994–2025) hei )Oct 15, 2025 24 search , ai What Is llms.txt, and Should You Care About It? ahr )Apr 22, 2025 23 ai , crawling Poisoning Well hey )Mar 31, 2025 22 ai , crawling , content Meet llms.txt, a Proposed Standard for AI Website Content Crawling sea )Mar 28, 2025 21 ai , crawling , scraping SEO jam +/htt )Dec 2, 2024 20 web-almanac , studies , research , metrics , seo , metadata , structured-data , amp , internationalization Why I Don’t Block AI Scrapers j9t )Aug 29, 2024 19 ai , scraping Google Quietly Launches New AI Crawler sea )Aug 21, 2024 18 google , ai , crawling Websites Are Blocking the Wrong AI Scrapers (Because AI Companies Keep Making New Ones) 404 )Jul 29, 2024 17 ai , scraping The Backlash Against AI Scraping Is Real and Measurable 404 )Jul 23, 2024 16 ai , scraping AI Unplugged: Rise (and Fall) of the Robots(.txt) Jul 8, 2024 15 ai , scraping Investigating Reddit’s robots.txt Cloaking Strategy Jul 4, 2024 14 scraping , web AI Companies Ignoring robots.txt mjt )Jun 24, 2024 13 ai , scraping ai.robots.txt cor )Mar 27, 2024 12 ai , crawling , scraping , tooling Go Ahead and Block AI Web Crawlers cor )Mar 2, 2024 11 crawling , scraping , ai The Text File That Runs the Internet dav /ver )Feb 14, 2024 10 crawling , scraping , ai , web Dark Visitors Nov 1, 2023 9 websites , ai , scraping Crawlers ada )Oct 2, 2023 8 crawling , ai Block the Bots That Feed “AI” Models by Scraping Your Website cla )Aug 23, 2023 7 scraping , ai OpenAI Launches Web Crawling GPTBot, Sparking Blocking Effort by Website Owners and Creators ven )Aug 8, 2023 6 ai , openai , crawling , scraping robots.txt Is 25 Years Old Jul 1, 2019 5 anniversaries , search Titles, “meta” Tags, “link” Tags, and Search Engine Robots Jul 13, 1997 4 html , metadata , seo robots.txt Validator (Merkle) max )3 tools , analysis , conformance , seo robots.txt Generator 2 tools , exploration , code-generation , seo robots.txt Validator (Logeix) 1 tools , analysis , conformance , seo 