How to Find a Website’s Public JSON Data Without Scraping Rendered HTML
JSON APIDevToolsweb scrapingbrowser automationdata extractionHTTP requests

How to Find a Website’s Public JSON Data Without Scraping Rendered HTML

wwebscraper.cloud Editorial Team
2026-11-06
6 min read

Learn how to inspect browser network requests, find public JSON data, test pagination, and choose JSON extraction over rendered HTML responsibly.

When a modern page displays a list, chart, or product catalog, the browser may be receiving structured JSON behind the scenes. This checklist shows how to find that data with browser DevTools, decide whether an endpoint is suitable for a permitted data workflow, and avoid scraping rendered HTML when a cleaner request is available.

Overview

A page can contain useful data in several places: the original HTML, embedded JSON, a JavaScript bundle, or an HTTP response returned after the page loads. The most reliable way to determine which one applies is to observe the browser’s network activity rather than guessing endpoint names.

This is not a method for bypassing access controls. Work only with data you are authorized to access, respect the site’s terms and technical restrictions, and avoid collecting personal or confidential information. A public response is not automatically permission for every use. For broader background, see How to Scrape a Website: A Practical Guide to HTML, JavaScript, and Structured Data.

The basic workflow is:

  1. Open the page and identify the data you need.
  2. Record the initial document request and inspect the Network panel.
  3. Filter requests by Fetch/XHR and reload the page.
  4. Find the response containing the relevant records.
  5. Replay the request carefully, then test pagination, filters, and error handling.
  6. Choose a stable, documented API where one exists; otherwise, use the smallest permitted extraction method.

Checklist by scenario

Scenario 1: The data appears after the page loads

Open browser DevTools, select Network, enable the request log, and reload the page. In Chrome- and Chromium-based browsers, the request-type filters commonly include Fetch/XHR; other browsers use similar labels. Clear the log before reloading so that older requests do not obscure the result.

Look for requests whose response preview contains the visible records. Useful clues include paths containing terms such as api, data, search, query, items, or graphql, but names vary widely. Select a likely request and inspect:

  • Headers: request URL, method, query parameters, and request headers.
  • Payload: JSON or form data sent with POST, PUT, or GraphQL requests.
  • Preview and Response: the returned object, array, or error message.
  • Timing: whether the request occurs on page load, after a click, or during scrolling.

Use the page’s own controls while recording requests. Change a filter, move to the next page, or trigger a sort operation. The new request often reveals the parameters needed to reproduce the result.

Scenario 2: The page uses a search box, filter, or pagination

Start with one simple action, such as entering a distinctive search term. Compare the requests made before and after the action. You may find query parameters like q, page, limit, offset, sort, or filter. These are conventions, not guarantees, so confirm their meaning by changing one value at a time.

For numbered pagination, check whether the request uses a page number, offset, cursor, or a token returned by the previous response. Cursor-based pagination requires sequential requests and should not be treated like a simple page counter. Stop when the response indicates that there is no next page, rather than assuming a fixed maximum.

For a repeatable extraction workflow, save a small sample response and document the request method, parameters, expected fields, and pagination behavior. The web scraping data cleaning workflow can help with the next stage: normalizing and validating the returned data.

Scenario 3: The request requires headers or a session

Some endpoints work only when the browser sends ordinary context such as an Accept header, a language preference, or session cookies. In DevTools, use Copy as cURL or an equivalent option to create a diagnostic copy of the request. Review it before running it elsewhere, and remove secrets from anything you store or share.

A request that succeeds only after signing in, submitting a form, or receiving a short-lived token is not the same as an unauthenticated public endpoint. Do not attempt to defeat authentication or token controls. If you are authorized to access the account or system, use the documented API or an approved application integration where possible. For understanding token payloads without treating decoded claims as proof of validity, see JWT Decoder Online: How to Inspect Claims Safely and Verify Tokens.

Scenario 4: No useful Fetch/XHR request appears

Check the page source and the Elements panel for JSON-LD, script tags containing serialized state, or data attributes. A framework may embed an initial state object in the document and then request more data only after interaction. You can also inspect the document request’s response type and search its body for a distinctive visible value.

If the data is generated only in the browser, browser automation may be appropriate. Compare the trade-offs in Web Scraping with BeautifulSoup vs Playwright and the Playwright workflow for JavaScript-rendered websites.

What to double-check

QuestionAPI or JSON requestRendered HTML
Where is the data?Usually in a structured responseIn the final document or DOM
What must be reproduced?Method, URL, parameters, and permitted headersPage navigation and selector logic
Typical failure pointSchema, token, cursor, or rate changesLayout and selector changes
Best validationCompare fields and record countsCompare extracted text with the visible page

Before building on a JSON response, verify that it actually represents the page state you need. Check whether records are truncated, localized, sorted differently, or split across multiple requests. Inspect HTTP status codes and response headers, but do not infer authorization from a successful browser request alone.

Also identify the response schema. A list may be under items, results, data, or a nested path. Fields may be absent, null, or typed differently across records. Save raw responses separately from transformed output so you can diagnose changes later.

Common mistakes

  • Searching only the page source: client-rendered content may never appear in the initial HTML.
  • Guessing API URLs: endpoint naming is inconsistent; inspect an observed request instead.
  • Copying every browser header: unnecessary headers add fragility and may expose credentials. Keep only what the permitted workflow requires.
  • Ignoring request methods: a POST request may carry filters in its body even when the page looks like a simple search.
  • Assuming one response is complete: check pagination, lazy loading, and secondary detail requests.
  • Treating decoded tokens as verified: decoding reveals format and claims; it does not validate a token’s signature or authority.
  • Skipping data validation: compare a sample with the visible interface and test missing values, duplicates, dates, and URLs.
  • Overlooking operational limits: keep request volume reasonable, cache permitted results, and design a clear stop condition.

If you need to inspect or transform the captured response during development, a browser-based JSON formatter can make nested objects easier to review. Keep sensitive payloads out of tools you do not control, and redact credentials before sharing examples.

When to revisit

Revisit this checklist whenever the site changes its frontend, search controls, login flow, or data model. A workflow that worked during one seasonal planning cycle may fail when filters, pagination, or deployment infrastructure changes. Schedule a small validation run before an important reporting period rather than discovering a broken endpoint after collection begins.

At each review, repeat five checks:

  1. Confirm that the request still returns the intended records.
  2. Compare the response schema with the fields your code expects.
  3. Test the first page, a later page, an empty result, and an error response.
  4. Verify that authentication, retention, and permitted-use assumptions remain accurate.
  5. Record the date, sample request, sample response, and any required code changes.

If the endpoint has become unstable or access is clearly intended for applications, stop reverse-engineering the page and look for an official integration or a documented export. When browser behavior is the only approved route, keep automation narrow and observable. For scheduling repeat checks, the Cron Expression Builder and Tester provides a practical way to review timing expressions before deployment.

The durable lesson is simple: inspect what the browser actually requests, validate what comes back, and treat the network workflow as an implementation detail that must be reviewed whenever the site or your requirements change.

Related Topics

#JSON API#DevTools#web scraping#browser automation#data extraction#HTTP requests
w

webscraper.cloud Editorial Team

Technical Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.