How to Scrape JavaScript-Rendered Websites with Playwright: A Practical Workflow

Overview

When you scrape a JavaScript website, the browser may receive a minimal document first and then fetch products, listings, comments, or other records through client-side requests. A parser that only downloads the original HTML can therefore return an empty table or incomplete fields. Playwright solves this by controlling a real browser engine, allowing your scraper to observe the page after scripts have run.

The goal is not to reproduce every user action. A reliable dynamic website scraper should do the minimum necessary:

• Open the intended page and establish a predictable browser context.
• Wait for a meaningful page condition rather than using arbitrary delays.
• Locate records with resilient selectors.
• Extract and normalize fields into a consistent schema.
• Move through pagination without duplicating or skipping records.
• Save checkpoints and logs so a partial run can be inspected or resumed.

Before automating, check that you have permission to collect the information and that your request pattern is appropriate for the site. Keep the scope narrow, avoid collecting unnecessary personal data, and treat the site interface as an input that can change.

For background on choosing between HTML parsing, browser automation, and structured endpoints, see How to Scrape a Website: A Practical Guide to HTML, JavaScript, and Structured Data.

Step-by-step workflow

1. Define the output before writing selectors

Start with a small schema. For a listing page, this might be name, price, url, category, and collected_at. Decide which fields are required, how missing values should be represented, and whether URLs should be stored as absolute URLs. This prevents the scraper from collecting large amounts of page content that you will later discard.

2. Install Playwright and create a controlled context

In Python, install the package and browser binaries in the environment used for the job:

pip install playwright
playwright install chromium

A basic setup can use a headless browser for scheduled runs and a visible browser during development. Keep timeouts, locale, viewport, and user-agent decisions in configuration rather than scattering them through the code.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    context = browser.new_context(
        viewport={"width": 1440, "height": 900}
    )
    page = context.new_page()
    page.goto("https://example.com/catalog", wait_until="domcontentloaded")
    browser.close()

The same workflow is available in Node.js. Choose one language for the project and keep browser creation, extraction, and storage separate so each part can be tested independently.

3. Wait for a condition that represents usable data

Fixed sleeps are easy to write but difficult to maintain. A page may load faster or slower depending on the environment, and a timer does not confirm that the records are present. Prefer a locator-based wait:

page.goto(url, wait_until="domcontentloaded")
page.locator("article.product-card").first.wait_for(state="visible")

If the page renders after a particular API request, waiting for the relevant response can be more precise. Use this only when the request pattern is stable and the response is genuinely the source of the records:

with page.expect_response(lambda response: "/api/products" in response.url) as event:
    page.reload()
response = event.value

Do not assume that networkidle means the page is complete. Analytics, advertisements, and background polling can keep a page active. A visible record container, a heading, or a known empty-state message is usually a clearer completion signal.

4. Select records with stable locators

Avoid selectors based on generated class names, deep positional paths, or the current visual layout. Prefer semantic attributes, accessible roles, meaningful data attributes, and a small number of structural relationships. For example:

cards = page.locator("article[data-testid='product-card']")
for index in range(cards.count()):
    card = cards.nth(index)
    name = card.locator("[data-testid='product-name']").inner_text()
    link = card.locator("a").get_attribute("href")

If the site has no stable attributes, isolate the least fragile selector available and add a test fixture for it. A selector should identify the same field even when unrelated elements are added to the page.

5. Normalize fields as you extract them

Whitespace, currency symbols, line breaks, and relative links create avoidable cleanup work. Normalize text at the boundary, but do not silently alter values that require interpretation. Preserve the original text when a transformation could lose meaning.

from urllib.parse import urljoin

def clean_text(value):
    return " ".join(value.split()) if value else None

record = {
    "name": clean_text(name),
    "url": urljoin(page.url, link) if link else None,
}

For a broader export process covering text normalization, URL validation, and CSV or JSON output, use the Web Scraping Data Cleaning Workflow.

6. Handle pagination deliberately

Pagination can use a next button, numbered links, an offset parameter, or infinite scrolling. For a next button, collect the current page, determine whether the control is available and enabled, then click and wait for a change in the record set.

seen_urls = set()

while True:
    page.locator("article[data-testid='product-card']").first.wait_for()
    # Extract records here and add their URLs to seen_urls.

    next_button = page.locator("a[aria-label='Next page']")
    if next_button.count() == 0 or not next_button.is_enabled():
        break

    previous_url = page.url
    next_button.click()
    page.wait_for_function("oldUrl => location.href !== oldUrl", previous_url)

Use a maximum-page limit and a duplicate check. These safeguards help when a site loops back to an earlier page or when a disabled control remains in the DOM.

7. Add pacing, retries, and checkpoints

Browser automation is slower and more resource-intensive than a direct HTTP client, so make each page visit intentional. Add a modest delay between completed pages where appropriate, use bounded retries for transient failures, and write records incrementally instead of waiting until the entire run ends.

A checkpoint can contain the last successful page, the run identifier, and the number of records written. Save failures with the URL, exception type, and a screenshot or HTML snapshot when permitted. This turns a difficult rerun into a targeted repair.

Tools and handoffs

Playwright is best used as the browser layer, not as the whole data pipeline. A maintainable workflow usually has four handoffs:

1. Discovery: inspect the page, identify the record container, and determine whether data comes from rendered markup or a request.
2. Collection: use Playwright for navigation, interaction, waiting, and extraction.
3. Validation: check required fields, URL formats, duplicate keys, and expected record counts.
4. Storage: write newline-delimited JSON, CSV, or a database table with run metadata.

Use browser developer tools to inspect network requests and DOM changes while developing. A JSON formatter can make captured response bodies easier to inspect, while a URL encoder or decoder can help when query parameters contain filters or pagination state. The Online Developer Tools Cheat Sheet collects related utilities for these handoffs.

BeautifulSoup can still be useful after Playwright has rendered a page, especially when you want a familiar HTML parsing model. In many cases, however, Playwright locators are sufficient and reduce the need to pass large HTML snapshots between tools. Choose the smallest toolchain that produces verifiable output.

Quality checks

Test the scraper against a small, known sample before running it across many pages. Compare the extracted output with the visible page and verify:

• Every required field is populated when the source displays it.
• Relative links resolve to the expected host and path.
• Pagination produces no duplicate records or unexpected gaps.
• Empty states are distinguished from loading failures.
• Text is normalized consistently without changing meaningful content.
• Retries do not create duplicate rows.
• A browser or selector failure leaves enough diagnostic information to investigate.

Run a smoke test whenever selectors change. It should open one representative page, extract a small number of records, and assert basic conditions such as a non-empty name and a valid URL. Add a second fixture for an empty result page or a missing optional field so the scraper is tested against more than the happy path.

When output is used for SEO or publishing workflows, validate URLs and review duplicate or near-duplicate text before import. The Text Similarity Checker Guide explains where similarity review fits into a broader content process.

When to revisit

Revisit a Playwright scraper whenever its inputs change, not only when the job fails. A page redesign may preserve the visible content while changing attributes, pagination controls, or loading behavior. A browser update, a new authentication flow, or a changed API response can also affect timing and selectors.

Set a review routine around observable signals: a sudden drop in record count, a rise in missing fields, repeated navigation failures, new duplicate rates, or a growing number of retries. Keep a small sample of expected output so each run can be compared with a known baseline.

For a practical maintenance cycle, follow these steps:

1. Run the smoke test against a representative page.
2. Inspect the rendered DOM and network activity if results change.
3. Update the narrowest selector or wait condition that broke.
4. Repeat pagination and duplicate checks on a small range.
5. Record the change, affected fields, and date in the scraper notes.

Keep browser settings, selectors, schemas, and storage logic configurable where possible. That separation makes the workflow easier to update when tools or platform features change, while preserving the core process: render only what is needed, wait for evidence, extract stable fields, validate the result, and save a recoverable dataset.