Robots.txt, Terms of Service, and Rate Limits: A Practical Web Scraping Compliance Checklist

Overview

If you scrape websites long enough, you learn that technical success and operational safety are not the same thing. A script can return perfect HTML, paginate correctly, and still create avoidable risk if it ignores site rules, overloads endpoints, or captures data that the team has not thought through carefully.

This article gives you a reusable web scraping compliance checklist for three recurring questions:

• What does the site appear to allow or discourage technically?
• What do the published terms, usage rules, or access conditions say?
• How do you run the scraper in a way that minimizes disruption and avoids careless behavior?

It is not legal advice, and it should not replace internal review where legal, privacy, or security teams are involved. But it will help you create a better process than the common pattern of “the script worked in testing, so ship it.”

A good working model is to separate compliance review into four layers:

1. Access layer: Is the content public, authenticated, paywalled, or restricted by region, account type, or technical gate?
2. Rules layer: What does robots.txt scraping review show, and what do the terms of service or site policies say?
3. Impact layer: What request volume, concurrency, and retry behavior are acceptable for this target?
4. Data layer: What data are you collecting, storing, enriching, and redistributing?

That framework matters because many scraping problems are not caused by one dramatic mistake. They usually come from small omissions: no one checked whether search pages were disallowed, retries were too aggressive, session-based pages were scraped as if they were public, or raw personal data was copied into downstream systems without a clear reason.

Before you build a larger workflow, it also helps to choose the right stack. If you are deciding between lightweight requests, browser automation, or a full rendering workflow, see Playwright vs BeautifulSoup vs Selenium for Web Scraping: Which Stack Fits Your Use Case?. Tool choice affects how visible, heavy, and complex your scraper becomes.

Checklist by scenario

Use this section as a pre-flight review. The scenarios are intentionally practical, because the right compliance posture often depends on what you are scraping and how often you plan to do it.

Scenario 1: Public pages, low-frequency research crawl

This is the most common entry point: public product pages, blog archives, directory listings, or documentation pages collected for analysis or internal reference.

Checklist:

• Confirm the content is publicly accessible without login, payment, or technical circumvention.
• Review robots.txt for the relevant user-agent rules and the specific path patterns you plan to crawl.
• Read the site terms, acceptable use language, and any developer or API documentation.
• Check whether an official export, feed, sitemap, or API already exists.
• Start with conservative request rates and low concurrency.
• Identify a narrow scope: paths, fields, and depth limits.
• Store only the data you actually need for the stated purpose.
• Log timestamps, URL counts, and crawl settings so the run can be audited later.

This scenario is where many teams get overconfident. Because the pages are public, they assume every use is automatically low-risk. Public access does not remove the need to check site rules or act carefully. It simply means the access layer is less restrictive.

Scenario 2: High-volume crawl of public listings or search results

Large crawls create a different kind of risk. Even when the content is public, repeated requests to pagination, search endpoints, or filter combinations can become noisy quickly.

Checklist:

• Review robots.txt for listing, search, and query-string rules, not just content pages.
• Check whether the site exposes crawl traps such as endless filters, calendar pages, or faceted combinations.
• Cap crawl depth, pagination range, and duplicate URL variants.
• Use pacing, randomized delays within reasonable bounds, and strict concurrency limits.
• Implement backoff on 429, 403, 5xx, and timeout responses.
• Honor cache headers or conditional requests where appropriate.
• Schedule runs during off-peak periods if the site appears capacity-sensitive.
• Monitor error rates and stop automatically if the target begins to struggle.

If your target involves next buttons, infinite scroll, or API-backed listings, it is worth reviewing Web Scraping Pagination Patterns: Next Buttons, Infinite Scroll, Load More, and APIs. Pagination design changes both scraper complexity and site impact.

Scenario 3: Authenticated area or account-based access

This is where your review should become stricter. If a workflow requires login, membership, or any account-level access, treat that as a separate category from public scraping.

Checklist:

• Verify who owns the account and whether the access is authorized for the intended purpose.
• Review the terms and any account-specific restrictions closely.
• Do not assume normal browser visibility equals permission to automate extraction.
• Minimize stored session material and secure credentials properly.
• Avoid scraping any data outside the exact scope the account holder expects.
• Review privacy implications before collecting user-generated or account-linked data.
• Document why automation is necessary instead of a manual export or official endpoint.

As a rule of thumb, authenticated scraping should trigger a more formal internal review than scraping public pages.

Scenario 4: JavaScript-heavy sites and browser automation

Modern sites often require browser rendering to reach the final DOM or the underlying API calls. That does not change the compliance questions, but it can increase your operational footprint because browser sessions are heavier than plain HTTP requests.

Checklist:

• Confirm that browser automation is necessary rather than simply convenient.
• Scope the session tightly: disable unneeded assets when possible, reduce navigation churn, and avoid repeated full-page loads.
• Inspect network calls to see whether a documented API or structured endpoint exists.
• Respect the same robots, terms, and rate-limit considerations you would apply to raw HTTP scraping.
• Measure the real request volume generated by a browser, not just page count.

For dynamic targets, see How to Scrape JavaScript-Heavy Websites in 2026: Playwright, Puppeteer, and Browser Rendering Compared. Browser rendering can multiply requests in ways teams underestimate.

Scenario 5: Structured extraction for internal business workflows

Some of the most valuable scraping projects are routine business workflows: pulling tables, extracting vendor records, enriching lists, or collecting public signals for monitoring. These are often legitimate use cases, but they deserve disciplined guardrails because they tend to become recurring jobs.

Checklist:

• Define the business purpose in one sentence before you crawl.
• List the exact fields needed and ignore the rest.
• Set refresh frequency based on real need, not technical possibility.
• Add deduplication, change detection, and selective recrawling to reduce load.
• Decide how long raw HTML or screenshots will be retained.
• Document who can access the resulting dataset and how it may be reused.

For extraction patterns, these related guides may help: How to Scrape Tables From Websites Reliably and Best XPath and CSS Selector Strategies for Web Scraping. Better extraction logic often means fewer retries and less unnecessary crawling.

What to double-check

This is the part many teams skip because they assume the first pass was good enough. In practice, the second review catches the issues that become expensive later.

1. Robots.txt is only one input, not the whole answer

When people discuss robots.txt scraping, they often oversimplify it. Robots.txt can tell you a lot about crawl preferences and path-level restrictions, but it is not the only document that matters. You should double-check:

• Which user-agent rules apply to your crawler?
• Whether the paths you need are explicitly disallowed, allowed, or unstated.
• Whether query parameters, search pages, feeds, or API-like endpoints are covered.
• Whether the robots file changed recently enough to warrant a fresh review.

Do not read robots.txt once and assume the answer is permanent. For recurring jobs, treat it like a versioned input that may change.

2. Terms of service may be spread across several pages

Terms of service scraping review is not always a single checkbox. Depending on the site, relevant language may appear in a main terms page, acceptable use policy, developer policy, data license page, paid plan terms, or account signup flow. Double-check whether restrictions are split across documents.

Create a short internal summary with three plain-language answers:

• What kind of access are we using?
• What kind of data are we collecting?
• What restrictions or conditions appear relevant to this workflow?

This summary is more useful operationally than pasting long legal text into a ticket that nobody rereads.

3. Rate limits are broader than 429 responses

Scraping rate limits are not always declared directly, and many sites do not express them in a developer-friendly way. That means you should infer safe operating limits conservatively.

Double-check:

• Average requests per second and peak bursts
• Concurrency per domain, per route, and per session
• Retry strategy after temporary failures
• Whether browser rendering inflates request counts through scripts, images, or XHR calls
• Whether pagination or filtering creates request explosions

A useful operating principle is that a scraper should degrade gracefully. If the site slows down, your crawler should slow down faster.

4. Data sensitivity and downstream use

Even if collection is technically simple, downstream use may raise new questions. Double-check whether your dataset includes:

• Personal information
• User-generated content
• Contact details
• Location or behavioral indicators
• Content that will later be enriched, scored, or redistributed

The compliance review should cover not just collection, but storage, access, retention, and reuse. A scraping script can be modest while the data pipeline built around it becomes much more sensitive.

5. Scope creep

Many scraping projects start narrow and become broad through convenience. A team begins with one public table, then adds profile pages, then user comments, then historical snapshots. Recheck whether the current run is still within the original scope. If the answer is no, rerun the checklist as if this were a new project.

Common mistakes

The fastest way to improve your process is to recognize the patterns that cause avoidable trouble. These mistakes appear across small scripts and large pipelines alike.

Treating public data as permission for any use

Publicly accessible does not mean context-free. Teams often move too quickly from “this page loads in a browser” to “we can automate this at scale.” A better approach is to distinguish visibility from acceptable access patterns.

Checking robots.txt but not the terms

Some developers use robots.txt as the whole compliance review. Others ignore it and only read the terms. Both are incomplete. The practical approach is to read both, compare them to the exact paths and methods you plan to use, and note any ambiguity.

Running production volume with test assumptions

A scraper that looks harmless in development can become noisy in production. Ten pages manually tested in a browser is not equivalent to fifty thousand requests across several workers. Always re-estimate request count, concurrency, and retries at real scale.

Ignoring hidden request multipliers

Infinite scroll, prefetching, browser assets, embedded APIs, and repeated retries can multiply traffic far beyond the visible page count. If you use a browser automation stack, inspect the actual network behavior before scheduling recurring jobs.

Capturing more data than necessary

Overcollection creates operational and compliance drag. If your use case needs product name, price, and availability, do not also keep every review, image URL, and session token just because they were easy to capture.

No stop conditions

Good scrapers need brakes. Set stop conditions for repeated 429s, rising 5xx rates, login failures, layout shifts, and abnormal page growth. Without them, a routine job can continue hammering a site after conditions have clearly changed.

No written decision trail

The most common long-term failure is undocumented judgment. Six months later, nobody remembers why a path was approved, how often the job was meant to run, or whether anyone reviewed the terms page. A short checklist completed before launch is often enough to avoid this problem.

When to revisit

This checklist is most useful when it becomes a recurring habit instead of a one-time article you read and forget. Revisit your scraping review in the following situations:

• Before seasonal planning cycles: teams often expand monitoring, vendor research, or competitive data collection during planning windows. That is exactly when low-volume scripts become high-volume jobs.
• When workflows or tools change: moving from requests to Playwright, adding proxies, changing pagination logic, or introducing enrichment steps all justify a fresh review.
• When the target site changes: new robots rules, revised terms, redesigned navigation, or different anti-bot behavior should trigger reevaluation.
• When data use changes: internal analysis is different from external publication, lead enrichment, scoring, or redistribution.
• When ownership changes: if a different team inherits the scraper, make them confirm scope, rules, and rate settings instead of assuming the old defaults are still appropriate.

To make this practical, keep a short operating checklist with every recurring scraper:

1. Target domain and paths
2. Business purpose
3. Public or authenticated access
4. Robots.txt review date
5. Terms/policy review date
6. Request rate and concurrency limits
7. Retry and backoff behavior
8. Data fields collected
9. Retention and downstream use notes
10. Owner responsible for reevaluation

That simple record turns compliance from memory into process.

If you are building a larger scraping program, pair this checklist with technical discipline: reliable selector strategy, careful pagination handling, and the lightest tool that can do the job. Those choices reduce both maintenance burden and target impact. Useful follow-up reads include pagination patterns, selector strategies, and stack selection.

The goal is not perfection. It is repeatable judgment. Before you run the next crawl, ask: what are we accessing, what rules apply, what load will we generate, and what data do we truly need? If your team can answer those four questions clearly every time, your scraping workflow will already be in better shape than most.