| Meet llms.txt, a Proposed Standard for AI Website Content Crawling (sea) | | 38 |
| ai, crawling, robotstxt |
| Web Scraping With Cheerio in 2025 (api) | | 37 |
| guides, tooling |
| Web Scraping With Playwright | | 36 |
| playwright, typescript, youtube, functionality |
| Clean Up HTML Content for Retrieval-Augmented Generation With Readability.js (phi/dat) | | 35 |
| html, tooling, nodejs |
| How to Scrape Web Content for RAG With Readability.js (phi/dat) | | 34 |
| videos, how-tos, content, ai |
| llms-txt | | 33 |
| websites, ai, crawling |
| Why I Don’t Block AI Scrapers (j9t) | | 32 |
| ai, robotstxt |
| Websites Are Blocking the Wrong AI Scrapers (Because AI Companies Keep Making New Ones) (404) | | 31 |
| ai, robotstxt |
| The Backlash Against AI Scraping Is Real and Measurable (404) | | 30 |
| ai, robotstxt |
| AI Unplugged: Rise (and Fall) of the Robots(.txt) | | 29 |
| ai, robotstxt |
| Investigating Reddit’s robots.txt Cloaking Strategy | | 28 |
| robotstxt, web |
| Consent, LLM Scrapers, and Poisoning the Well (eri) | | 27 |
| ai, legal |
| AI Companies Ignoring robots.txt (mjt) | | 26 |
| ai, robotstxt |
| Let’s Build a Web Scraper in PHP and Python | | 25 |
| php, python |
| Who Should Block AI Bots? (moz) | | 24 |
| ai, seo |
| Blockin’ Bots (bee) | | 23 |
| ai, apache, configuration |
| ai.robots.txt (cor) | | 22 |
| ai, crawling, robotstxt, tooling |
| Go Ahead and Block AI Web Crawlers (cor) | | 21 |
| robotstxt, crawling, ai |
| The Text File That Runs the Internet (dav/ver) | | 20 |
| robotstxt, crawling, ai, web |
| Dark Visitors | | 19 |
| websites, ai, robotstxt |
| Personal-Scale Web Scraping for Fun and Profit | | 18 |
| javascript, functionality, optimization |
| Block the Bots That Feed “AI” Models by Scraping Your Website (cla) | | 17 |
| robotstxt, ai |
| OpenAI Launches Web Crawling GPTBot, Sparking Blocking Effort by Website Owners and Creators (ven) | | 16 |
| ai, openai, crawling, robotstxt |
| Puppeteer in Node.js: More Antipatterns to Avoid (app) | | 15 |
| nodejs, testing, anti-patterns, puppeteer |
| Scraping Single-Page Applications With Playwright (api) | | 14 |
| single-page-apps, playwright |
| Web Scraping—A Complete Guide | | 13 |
| guides |
| Sophisticated Web Scraping With Bright Data (cra) | | 12 |
| structured-data, apis |
| Web Scraping via JavaScript Runtime Heap Snapshots | | 11 |
| javascript, memory |
| Web Scraping Is Legal, U.S. Appeals Court Reaffirms (tec) | | 10 |
| legal |
| Web Scraping With JavaScript and Node.js | | 9 |
| javascript, nodejs |
| Web Crawling vs. Web Scraping | | 8 |
| crawling, comparisons, terminology |
| Web Crawler vs. Web Scraper: The Differences | | 7 |
| crawling, comparisons, terminology |
| No Need to Protect Your Website From Scraping: 8 Reasons | | 6 |
| web, seo, legal |
| The Ultimate Guide to Building Scalable Web Scrapers With Scrapy (sma) | | 5 |
| guides, tooling, python |
| Web Scraping With Node.js (sma) | | 4 |
| nodejs, javascript |
| Using .htaccess to Prevent Web Scraping | | 3 |
| servers, apache |
| The Rise of Web Bots and Fall in Human Traffic (cra) | | 2 |
| web, spam, traffic, metrics |
| Web Scraping in Node.js | | 1 |
| nodejs |
