| What Is llms.txt, and Should You Care About It? (ahr) | | 22 |
| ai, crawling |
| Poisoning Well (hey) | | 21 |
| ai, crawling, content |
| Meet llms.txt, a Proposed Standard for AI Website Content Crawling (sea) | | 20 |
| ai, crawling, scraping |
| SEO (jam+/htt) | | 19 |
| web-almanac, studies, research, metrics, seo, metadata, structured-data, amp, internationalization |
| Why I Don’t Block AI Scrapers (j9t) | | 18 |
| ai, scraping |
| Google Quietly Launches New AI Crawler (sea) | | 17 |
| google, ai, crawling |
| Websites Are Blocking the Wrong AI Scrapers (Because AI Companies Keep Making New Ones) (404) | | 16 |
| ai, scraping |
| The Backlash Against AI Scraping Is Real and Measurable (404) | | 15 |
| ai, scraping |
| AI Unplugged: Rise (and Fall) of the Robots(.txt) | | 14 |
| ai, scraping |
| Investigating Reddit’s robots.txt Cloaking Strategy | | 13 |
| scraping, web |
| AI Companies Ignoring robots.txt (mjt) | | 12 |
| ai, scraping |
| ai.robots.txt (cor) | | 11 |
| ai, crawling, scraping, tooling |
| Go Ahead and Block AI Web Crawlers (cor) | | 10 |
| crawling, scraping, ai |
| The Text File That Runs the Internet (dav/ver) | | 9 |
| crawling, scraping, ai, web |
| Dark Visitors | | 8 |
| websites, ai, scraping |
| Crawlers (ada) | | 7 |
| crawling, ai |
| Block the Bots That Feed “AI” Models by Scraping Your Website (cla) | | 6 |
| scraping, ai |
| OpenAI Launches Web Crawling GPTBot, Sparking Blocking Effort by Website Owners and Creators (ven) | | 5 |
| ai, openai, crawling, scraping |
| Titles, “meta” Tags, “link” Tags, and Search Engine Robots | | 4 |
| html, metadata, seo |
| robots.txt Validator (Logeix) | | 3 |
| tools, analysis, conformance, seo |
| robots.txt Generator | | 2 |
| tools, exploration, code-generation, seo |
| robots.txt Validator (Merkle) (max) | | 1 |
| tools, analysis, conformance, seo |
