How to Scrape a Website: A Practical Guide to HTML, JavaScript, and Structured Data

Overview

Web scraping is the process of retrieving information from web pages and turning it into a usable format such as JSON, CSV, or a database record. The right approach depends less on the size of the task than on how the target page delivers its content.

For a mostly static HTML page, a simple HTTP request and HTML parser may be enough. Python with BeautifulSoup is a practical choice when you need repeatable scripts, custom cleaning, or integration with another data workflow. A browser-based web scraper tool can be useful when you want to inspect a page, select elements, test an extraction quickly, or avoid setting up a local development environment. Playwright is better suited to pages that require a real browser, including pages whose content appears only after JavaScript runs.

Before extracting anything, define the output rather than starting with selectors. Write down the fields you need, such as title, URL, price label, publication date, author, or product identifier. This prevents a scraper from collecting large amounts of incidental page content that is difficult to validate later.

Use this basic decision guide:

• Static HTML: Start with a request and BeautifulSoup, or a browser-based extraction tool for a quick one-off task.
• JavaScript-rendered content: Use Playwright or another browser automation workflow when the required elements are not present in the initial HTML.
• Repeated production jobs: Prefer a tested script with logging, retries, validation, and controlled output.
• Exploration or prototyping: Use browser developer tools or a web scraper tool to inspect the page structure before writing code.

Scraping should be conducted responsibly. Check the site’s published guidance, access only the data you need, avoid collecting sensitive information without a clear legitimate purpose, and keep request rates conservative. If the data will support a business process, document the source, collection time, and intended use.

Checklist by scenario

Scenario 1: You need a few fields from static HTML

Begin by viewing the page source or inspecting the element in browser developer tools. Confirm that the text you need is present in the returned HTML rather than inserted later by JavaScript. Look for stable attributes such as a meaningful class, an ID, a data attribute, or a semantic element. Avoid relying on a long chain of positional selectors such as “the third div inside the second section.”

A small BeautifulSoup example might look like this:

import requests
from bs4 import BeautifulSoup

url = "https://example.com/articles"
response = requests.get(url, timeout=20)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
records = []

for item in soup.select("article.card"):
    title = item.select_one("h2")
    link = item.select_one("a")
    if title and link:
        records.append({
            "title": title.get_text(" ", strip=True),
            "url": link.get("href")
        })

This beautifulsoup example is intentionally small. In a real Python web scraping guide, you would also normalize relative URLs, handle missing fields, record failures, and save the result in a defined schema.

Scenario 2: You want to test an extraction in the browser

A browser-based web scraper tool is useful during discovery. Load the target URL, inspect the page visually, identify the repeated record container, and map each field to a selector. Test the extraction on several records rather than accepting the first successful result. Check whether the tool is reading visible text, attributes such as href, or embedded data.

Use a sample output with a few records and verify it manually. This catches common problems such as selecting navigation links instead of article links, capturing hidden duplicate text, or treating a label as the actual value.

Scenario 3: The page depends on JavaScript

If the initial HTML does not contain the data, inspect the page’s network activity and rendered DOM. The page may load data through an API request, or it may build the required elements after scripts execute. If an accessible documented endpoint provides the data you need, it may be more stable than scraping the rendered interface. If browser rendering is required, use Playwright.

A minimal Playwright pattern is:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://example.com/catalog", wait_until="domcontentloaded")
    page.locator("article.card").first.wait_for()

    records = []
    for card in page.locator("article.card").all():
        records.append({
            "title": card.locator("h2").inner_text().strip(),
            "url": card.locator("a").get_attribute("href")
        })

    browser.close()

Do not assume that a fixed sleep is a reliable wait strategy. Prefer a condition tied to the element or state you actually need. For a deeper workflow covering browser automation, selectors, waits, and reliability, see the Playwright web scraping guide.

Scenario 4: You need pagination or repeated collection

Decide how pagination works before implementing it. It may use numbered links, a next-page URL, an offset parameter, or a “load more” control. Set a clear stopping condition: no next link, no new records, a known page limit, or a repeated URL. Deduplicate by a stable key such as a canonical URL or source identifier, not by title alone.

Save intermediate results when a job spans many pages. A failed run should not require starting from the beginning, and every record should retain useful provenance such as source URL and collection timestamp.

What to double-check

• Selector stability: Test selectors against several pages, record types, and content variations. Prefer semantic or data-specific attributes over styling classes that may change with a redesign.
• Rendered versus source content: Compare the initial HTML with the browser’s rendered DOM. Content visible in a browser is not always available to a basic HTTP request.
• URL handling: Convert relative links to absolute URLs, preserve meaningful query parameters, and remove tracking parameters only when your workflow can do so safely. A canonical tag checker can help review duplicate or parameterized URL behavior after collection.
• Text quality: Strip excess whitespace, decode entities correctly, preserve meaningful line breaks, and distinguish an absent value from an empty value.
• Data types: Keep dates, numbers, currencies, and identifiers in predictable fields. Store the original text when parsing could be ambiguous.
• Pagination completeness: Compare the number of pages visited with the number of records returned. A successful browser run can still produce incomplete data if a selector stops matching.
• Output validation: Check required fields, URL formats, duplicate keys, and unexpected empty values before handing the data to another system. A JSON formatter can make a sample export easier to inspect.
• Access and privacy: Confirm that your collection approach is appropriate for the source and use case. Avoid authentication bypasses, unnecessary personal data, and aggressive request patterns.

Keep a small fixture of previously collected HTML or rendered output for testing. When the target page changes, you can compare the new extraction with a known sample instead of debugging against a moving target.

Common mistakes

Choosing a browser when a request is enough

Full browser automation adds setup, execution time, and more failure points. Use it because the page requires rendering or interaction, not simply because it is available.

Using a request against a JavaScript application

If the response contains an application shell but not the records, BeautifulSoup cannot extract content that was never returned. Inspect the network requests or switch to a browser workflow after confirming the limitation.

Writing selectors tied to presentation

Classes used only for layout or visual styling are likely to change. Look for stable attributes and include selector tests in your workflow.

Ignoring missing fields

Calling a text method on a missing element can stop an entire run. Treat optional fields as optional, log missing required fields, and continue only when the resulting record remains useful.

Assuming one successful page proves the scraper works

Test empty states, long titles, missing images, pagination boundaries, regional variations, and pages with unusual markup. A scraper is an extraction system, not just a selector.

Skipping post-processing

Raw extraction often contains duplicate URLs, navigation text, inconsistent dates, and escaped characters. Clean and validate data before exporting it. If the output feeds an SEO workflow, review canonical URLs, sitemap inclusion, structured data, and snippets separately rather than treating scraped fields as automatically publication-ready. Related checks include the sitemap validator guide and schema markup validator guide.

When to revisit

Revisit your scraping checklist before seasonal planning cycles, before restarting a dormant collection job, and whenever the target site changes its templates, navigation, rendering framework, or URL structure. Also review the workflow when you change from a manual browser-based task to scheduled automation, or when an output begins feeding a database, dashboard, or publishing process.

Schedule a practical review around these signals:

1. Record counts change unexpectedly.
2. Required fields become empty or contain navigation text.
3. HTTP responses succeed but rendered content is missing.
4. Duplicate URLs or records increase.
5. A browser, Python package, automation library, or deployment environment changes.
6. The purpose or sensitivity of the collected data changes.

At each review, run a small sample, compare it with a saved fixture, validate the exported JSON or CSV, and inspect a few source pages manually. Update selectors, waits, deduplication rules, and documentation together. If you are building a new browser automation workflow, pair this checklist with the text diff checker guide to compare output changes and the regex tester guide when validating URLs or structured text.

The most dependable approach to how to scrape a website is incremental: identify the smallest useful dataset, test the delivery method, validate a representative sample, and only then automate the full collection. That process keeps a web scraping tutorial practical long after the page, browser, or workflow has changed.