| Crawling December: CDNs and Crawling (by+/via) | | 10 |
| seo, content-delivery |
| Google Quietly Launches New AI Crawler (by/via) | | 9 |
| google, ai, robotstxt |
| Go Ahead and Block AI Web Crawlers (by) | | 8 |
| robotstxt, scraping, ai |
| The Text File That Runs the Internet (by/via) | | 7 |
| robotstxt, scraping, ai, web |
| Crawlers (by) | | 6 |
| robotstxt, ai |
| OpenAI Launches Web Crawling GPTBot, Sparking Blocking Effort by Website Owners and Creators (via) | | 5 |
| ai, scraping, robotstxt |
| OpenAI’s ChatGPT New Web Crawler—GPTBot (by/via) | | 4 |
| ai, seo |
| Web Crawling vs. Web Scraping (via) | | 3 |
| scraping, comparisons, terminology |
| Web Crawler vs. Web Scraper: The Differences (by) | | 2 |
| scraping, comparisons, terminology |
| W3C Unveils a Cure for Web Crawl | | 1 |
| w3c, performance, protocols, http |
