# Web Scraping API

Scrape any website into clean Markdown and HTML, with the page title, status code, final URL after redirects and a clear error when a page fails. Scrape a list of URLs, or search by keyword and scrape the top results in one call.

**Pricing:** $0.005 per page

**Endpoint:** `POST /v1/data/web/fetch/run`

**Auth:** `Authorization: Bearer mk_live_...`

## Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `urls` | array | No | One or more urls. One request reads them all. |
| `query` | string | No | Keyword to search the web for. The top results are fetched and converted to Markdown. |
| `maxResults` | integer | No | How many pages to return in total. Defaults to 10; use 0 for all. |

## Example

`?wait=true` holds the request open until the run finishes (up to 60s) and returns the rows inline: one request, no polling.

```bash
curl -X POST "https://api.mindcase.co/v1/data/web/fetch/run?wait=true" \
  -H "Authorization: Bearer mk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"params":{}}'
```

```python
import requests

resp = requests.post(
    "https://api.mindcase.co/v1/data/web/fetch/run",
    params={"wait": "true"},
    headers={"Authorization": "Bearer mk_live_YOUR_API_KEY"},
    json={"params": {}},
)
rows = resp.json()["data"]
```

## Longer runs

If the run is still going at the 60s ceiling you get back `{ job_id, status: "running" }` instead of rows (same if the result is over 100 rows; those come back `truncated`). Poll with that `job_id`:

```bash
# 1) check status
curl https://api.mindcase.co/v1/jobs/JOB_ID \
  -H "Authorization: Bearer mk_live_YOUR_API_KEY"

# 2) when status == completed, fetch the rows
curl https://api.mindcase.co/v1/jobs/JOB_ID/results \
  -H "Authorization: Bearer mk_live_YOUR_API_KEY"
```

Full API reference (auth, jobs, balance, SDKs, MCP): https://mindcase.co/skills.md

## Response columns

| Field | Display name | Type |
|-------|--------------|------|
| `url` | URL | text |
| `query` | Query | text |
| `rank` | Rank | number |
| `finalUrl` | Final URL | text |
| `statusCode` | Status Code | integer |
| `fetchStatus` | Fetch Status | text |
| `title` | Title | text |
| `domain` | Domain | text |
| `description` | Description | text |
| `snippet` | Snippet | text |
| `publishedDate` | Published Date | text |
| `wordCount` | Word Count | number |
| `markdown` | Markdown | text |
| `html` | Cleaned HTML | text |
| `htmlLength` | Html Length | integer |
| `meta` | Meta | object |
| `errorType` | Error Type | text |
| `errorMessage` | Error Message | text |