# Scrape multiple URLs in one request with Retriever AI's API

Retriever AI's Scrape API loads several pages in one call, and the tabs come back in the order you sent the URLs. Each call costs $0.05, however many URLs it carries. Send the list as `urls` to `POST https://agents.rtrvr.ai/zcj/tl2oxz0cmsfd/scrape`, then pay the 402 price quote and retry as [the buyer skill](https://agents.rtrvr.ai/zcj/tl2oxz0cmsfd/SKILL.md) describes, with no account or API key.

## At a glance

- **Endpoint:** `POST https://agents.rtrvr.ai/zcj/tl2oxz0cmsfd/scrape` with `Content-Type: application/json`
- **Body:** `urls` (required), an array of at least one absolute URL. Optional: `settings.extractionConfig`, `response.inlineOutputMaxBytes`, `webhooks`.
- **Price:** $0.05 per call, whatever the number of URLs. `usageData` in the response is informational and is not what you are billed.
- **Payment:** a 402 quote, paid from a wallet and retried. A card pays a single call only at $0.50 or more.
- **Response:** `tabs`, one entry per input URL in input order, each with `url`, `title`, `status`, `tree` and `elementLinkRecord`
- **Large results:** above `inlineOutputMaxBytes` (default `1048576`), tabs hold previews and `metadata.responseRef` points to the full payload
- **Full purchase flow:** [the buyer skill](https://agents.rtrvr.ai/zcj/tl2oxz0cmsfd/SKILL.md)

## Scrape a list of URLs in one call

1. Send the request without payment:

   ```bash
   curl -i -X POST https://agents.rtrvr.ai/zcj/tl2oxz0cmsfd/scrape \
     -H 'Content-Type: application/json' \
     -d '{"urls": ["https://example.com", "https://www.iana.org/help/example-domains"]}'
   ```

   The storefront returns `402 Payment Required`. The body carries `payment.amountUsd`, the `plan` and a `protocols` block.

2. Pay the 402 and send the identical request again with the proof attached. [The buyer skill](https://agents.rtrvr.ai/zcj/tl2oxz0cmsfd/SKILL.md) covers settling it on each rail.

3. Read `tabs` in order. `tabs[0]` belongs to the first URL you sent, and `tabs[1]` belongs to the second. A two-URL call returns this, trimmed to its first tab:

   ```json
   {
     "success": true,
     "tabs": [
       {
         "requestedUrl": "https://example.com/",
         "url": "https://example.com/",
         "title": "Example Domain",
         "contentType": "text/html",
         "status": "success",
         "tree": "[textNode] This domain is for use in documentation examples without needing perm…",
         "elementLinkRecord": {"8": "https://iana.org/[path]"}
       },
       "… 1 more"
     ],
     "outputTooLarge": false,
     "inlineLimitBytes": 1048576,
     "metadata": {"webhooks": {"requested": 0, "attempted": 0}}
   }
   ```

   That call paid $0.05 for both pages. Each id in `elementLinkRecord` matches an element in `tree` and maps to that link's target.

4. Check each tab's `status`. A tab with the value `error` carries the reason in its `error` field.

## Set parallel loads and the time budget

These settings go in `settings.extractionConfig`, as per-call overrides merged over the stored profile.

| To control | Set | Value |
| --- | --- | --- |
| How many URLs load at once | `maxParallelTabs` | integer |
| Budget for the whole call | `totalBudgetMs` | integer |
| Text only, no tree or link record | `onlyTextContent` | `true` |
| Skip the scroll pass for lazy-loaded content | `disableAutoScroll` | `true` |
| Extra wait after load, before extraction | `pageLoadDelay` | number |

```json
{
  "urls": ["https://example.com", "https://www.iana.org/help/example-domains"],
  "settings": {"extractionConfig": {"maxParallelTabs": 2, "totalBudgetMs": 60000}}
}
```

## Read large results

`response.inlineOutputMaxBytes` caps the inline JSON. Its default is `1048576` bytes. A larger result returns per-tab previews plus `metadata.responseRef`, the storage reference for the full payload. You can raise or lower the cap for each call:

```json
{"urls": ["https://example.com"], "response": {"inlineOutputMaxBytes": 262144}}
```

## Receive results by webhook

`webhooks` is a list of endpoints. Each one receives the result as a JSON POST when the scrape succeeds, fails or is cancelled.

```json
{
  "urls": ["https://example.com", "https://www.iana.org/help/example-domains"],
  "webhooks": [
    {
      "url": "https://hooks.example.com/rtrvr",
      "events": ["rtrvr.scrape.succeeded", "rtrvr.scrape.failed", "rtrvr.scrape.cancelled"],
      "secret": "<your HMAC secret>",
      "retry": {"mode": "default"},
      "timeoutMs": 8000
    }
  ]
}
```

- `url` is required and must be an HTTPS endpoint.
- `events` defaults to all three events.
- `secret` adds an `X-Rtrvr-Signature` header to every delivery.
- `auth` sends a `bearer` token or `basic` credentials to your endpoint, and `headers` adds extra headers to each delivery.
- `retry.mode` set to `default` retries with backoff. Set to `none`, it delivers once.
- `timeoutMs` defaults to `8000`.

The response metadata includes a `webhooks` object with `requested` and `attempted` counts.

## What is charged

- Each call is billed as one request at $0.05, whatever the number of URLs.
- A payment is tied to the request it was made for. A different request needs a fresh 402.
- The output is raw page data. No language model runs on it, so extraction is left to your own pipeline.

## Errors

| Code | Meaning | Action |
| --- | --- | --- |
| `402` with a `payment` block | The price quote | Pay it and retry the same request, or stop |
| `invalid_request` (400) | The body is malformed | Fix the JSON. `urls` must be an array of absolute URLs. |
| `payment_request_mismatch` (409) | A payment was reused for a different request | Request a fresh 402 and settle that one |
| `settlement_failed` | The payment did not settle | Fix the wallet, then send the request again for a fresh 402 |
| `payment_provider_unavailable` | A transient payment-provider failure | Retry unchanged, with backoff |
| Tab `status` is `error` | That URL did not load | Read the tab's `error` field |

## Frequently asked questions

### Does a call with more URLs cost more?

No. Every call is billed as one request at $0.05, whether `urls` holds one address or many. The `usageData` object in the response reports Retriever AI's infrastructure usage and is informational only.

### Do the tabs come back in the same order as the URLs?

Yes. `tabs` has one entry per input URL, in input order. Each entry also carries `url`, so you can match a tab to its page directly.

### How many pages load at the same time?

`settings.extractionConfig.maxParallelTabs` sets how many URLs load at once. `totalBudgetMs` sets the budget for the whole call.

### Can the scrape result be sent to a webhook?

Yes. Add a `webhooks` list. Each endpoint receives the result on success, failure or cancellation, and setting `secret` adds an `X-Rtrvr-Signature` header to each delivery.

### Do I need an API key or an account to scrape a list of URLs?

No. Each call is paid at the request by settling its 402 over x402 or MPP. Neither registration nor an upstream key is needed.

## Keep reading

- [Buyer skill](https://agents.rtrvr.ai/zcj/tl2oxz0cmsfd/SKILL.md): the full purchase flow
- [Scrape API reference](https://agents.rtrvr.ai/zcj/tl2oxz0cmsfd/services/scrape/openapi.json): every request and response field
- [Payment](https://agents.rtrvr.ai/zcj/tl2oxz0cmsfd/payment.md): the 402 and each rail
- [Error reference](https://agents.rtrvr.ai/zcj/tl2oxz0cmsfd/errors.md): every error code and the action for it

---

Updated 2026-10-28. All Retriever AI guides: https://agents.rtrvr.ai/guides.md. Storefront: https://agents.rtrvr.ai/zcj/tl2oxz0cmsfd/llms.txt.