# Read any web page as text and links with Retriever AI's Scrape API

Retriever AI's Scrape API loads a web page in a real browser and returns it the way an agent reads it: the text a visitor sees, an accessibility tree that gives each link a numeric id, and a lookup from those ids to where each link points. Each call costs $0.05, and a call can carry one URL or several. Send `POST https://agents.rtrvr.ai/zcj/fg2kv4xo47s6/scrape` with a `urls` list, pay the 402 price quote over x402 or MPP, and read one tab per URL in the response.

## At a glance

- **Endpoint:** `POST https://agents.rtrvr.ai/zcj/fg2kv4xo47s6/scrape`
- **Body:** `{"urls": ["https://example.com"]}`. `urls` is required and needs at least one absolute URL. Optional `settings` change what comes back.
- **Price:** $0.05, a flat rate for each call
- **Payment:** a 402 price quote that you pay from a wallet and retry. No sign-up, account or API key.
- **Response:** `tabs`, one per input URL in input order, each with `title`, `status`, `tree` and `elementLinkRecord` (with `onlyTextContent`, the text in `content` instead)
- **What runs:** real browsers behind rotating proxies, with no language model on the output
- **Full reference:** [the Scrape API reference](https://agents.rtrvr.ai/zcj/fg2kv4xo47s6/services/scrape/openapi.json)
- **Full purchase flow:** [the buyer skill](https://agents.rtrvr.ai/zcj/fg2kv4xo47s6/SKILL.md)

## What Scrape is for

Because each page loads in a real browser behind rotating proxies, sites that throttle or refuse plain HTTP requests still return their pages. No language model runs on the output. What you receive is raw page data for your own extraction, search or retrieval pipeline.

## Scrape a web page

1. Send the request without payment:

   ```bash
   curl -si -X POST https://agents.rtrvr.ai/zcj/fg2kv4xo47s6/scrape \
     -H 'Content-Type: application/json' \
     -d '{"urls": ["https://example.com"]}'
   ```

   The storefront answers `402`. The body carries `payment.amountUsd`, the `plan` and a `protocols` block with a challenge for each rail. The x402 challenge also arrives in the `payment-required` header and the MPP challenge in `www-authenticate`.

2. Pay the challenge and retry the same request ([the buyer skill](https://agents.rtrvr.ai/zcj/fg2kv4xo47s6/SKILL.md) has the steps). The example request paid $0.05 and returned in 17.2 seconds.

3. Read the `200` response. For example.com it has this shape, with `tree` shortened here:

   ```json
   {
     "success": true,
     "tabs": [
       {
         "observation": {"source": "accessibility_tree", "complete": true, "partial": false},
         "requestedUrl": "https://example.com/",
         "url": "https://example.com/",
         "tabId": 311688750,
         "title": "Example Domain",
         "contentType": "text/html",
         "status": "success",
         "tree": "[textNode] This domain is for use in documentation examples without needing perm…",
         "elementLinkRecord": {"264": "https://iana.org/[path]"}
       }
     ],
     "usageData": {"proxyPageLoads": 1, "usingBillableProxy": true},
     "egress": {"country": "US", "timezone": "America/New_York"},
     "outputTooLarge": false
   }
   ```

## Read a tab

- `tree` is the accessibility tree as a string. The page's visible text appears in it as text nodes, and every link carries a numeric id.
- With `onlyTextContent` set, the tab returns the extracted visible text in `content` and omits `tree` and `elementLinkRecord`.
- `elementLinkRecord` maps each link id to its target URL. To follow a link you found in the tree, look up its id here. In the example, id `264` maps to an iana.org address.
- `status` is `success` or `error`. An `error` tab carries an `error` field.
- `observation` describes the read, with `complete` and `partial` flags.
- `usageData` reports infrastructure usage on Retriever AI's side. It is informational, and the storefront bills one request per call.

## Change what a call returns

Settings go in the request body next to `urls`.

| To | Set | Value |
| --- | --- | --- |
| Get text only, without the tree and link map | `settings.extractionConfig.onlyTextContent` | `true` |
| Skip the scroll pass that triggers lazy-loaded content | `settings.extractionConfig.disableAutoScroll` | `true` |
| Wait longer after load before extraction | `settings.extractionConfig.pageLoadDelay` | a number |
| Control how many URLs load at once | `settings.extractionConfig.maxParallelTabs` | an integer |
| Use Retriever AI's rotating proxies | `settings.proxyConfig.mode` | `default` |
| Load from a place | `settings.proxyConfig.location` | `country`, `region`, `city` |
| Use your own proxies | `settings.proxyConfig.mode` and `customProxies` | `custom` |
| Receive the result at an HTTPS endpoint | `webhooks` | a list of objects with `url` |

```json
{"urls": ["https://example.com", "https://example.org"], "settings": {"extractionConfig": {"onlyTextContent": true, "maxParallelTabs": 2}}}
```

## Cost per call

Each call is billed as one request at a flat rate.

| Run | Cost |
| --- | --- |
| One call with one URL | $0.05 |
| Ten calls with one URL each | $0.50 |

## Limits

- Scrape returns page data only. It does not summarize, extract fields or run a model on the page.
- Scrape is sold pay as you go, one 402 per call.
- A card pays a single call only at $0.50 or more, so pay these calls from a wallet.
- The inline response is capped by `response.inlineOutputMaxBytes`, which defaults to 1048576 bytes. A larger result returns per-tab previews and `metadata.responseRef` for the full payload, with `outputTooLarge` set.
- Paying the 402 challenge is what completes the purchase.

## Errors

After a paid call is forwarded, the response carries Retriever AI's own status and body.

| Response | Meaning | Action |
| --- | --- | --- |
| `402` with a `payment` block and no `reason` | The price quote for this call | Pay and retry the same request, or stop |
| `settlement_failed`, `settlementReason: insufficient_funds` | The payment did not settle | Fund the wallet, then send the request again for a fresh 402 |
| `payment_request_mismatch` | A payment was reused for a different request | Request a fresh 402 for the new body |
| `400 invalid_request` | The body is malformed | Fix the JSON. `urls` needs at least one absolute URL |
| `400 payg_not_purchasable` | You tried to buy the Pay per request plan | Do not buy it. Call `/scrape` directly |
| Tab `status: "error"` | That URL did not load | Read the tab's `error` field |

## Frequently asked questions

### Do I need an account or API key to use the Scrape API?

No. Each call is paid on its own by settling its 402 over x402 or MPP. The signed payment is all the storefront needs, so there is no sign-up, identity registration or key for pay-as-you-go calls.

### Does Scrape return a page as a browser renders it?

Yes. Each URL loads in a real browser in Retriever AI's cloud browser cluster, and a scroll pass triggers lazy-loaded content before extraction. If content appears late, set `pageLoadDelay`. If the scroll pass is not needed, set `disableAutoScroll` to `true`.

### How do I get only the text of a web page?

Set `settings.extractionConfig.onlyTextContent` to `true`. The tab then returns extracted text only and omits `tree` and `elementLinkRecord`. The price per call stays the same.

### How much does it cost to scrape several pages?

$0.05 per call, since each call is billed as one request at a flat rate. A call can carry several URLs in `urls`, and the tabs come back in the order you sent them.

### Does a language model process the scraped output?

No. No model runs on the way out. The response is the raw text, accessibility tree and link map, ready for your own extraction or retrieval.

## Keep reading

- [Buyer skill](https://agents.rtrvr.ai/zcj/fg2kv4xo47s6/SKILL.md): the full purchase flow
- [How to pay](https://agents.rtrvr.ai/zcj/fg2kv4xo47s6/payment.md): the 402 and each rail
- [Scrape API reference](https://agents.rtrvr.ai/zcj/fg2kv4xo47s6/services/scrape/openapi.json): every request and response field
- [Error reference](https://agents.rtrvr.ai/zcj/fg2kv4xo47s6/errors.md): every error code and what to do
- [Catalog](https://agents.rtrvr.ai/zcj/fg2kv4xo47s6/manifest.json): live services and prices

---

Updated 2026-10-28. All Retriever AI guides: https://agents.rtrvr.ai/guides.md. Storefront: https://agents.rtrvr.ai/zcj/fg2kv4xo47s6/llms.txt.