# Web Scraping API

Scrape any web page or PDF live into clean Markdown with every link kept, plus title, description, author, published date, word count and the cleaned HTML. Pages that cannot be scraped come back with a typed error and are not charged. Scrape a list of URLs, or search by keyword and scrape the top results.

**Pricing:** $0.0025 per page

**Endpoint:** `POST /v1/data/web/scrape/run`

**Auth:** `Authorization: Bearer mk_live_...`

## Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `urls` | array | No | Web pages or PDF links to scrape, up to 1,000 per request. Failed pages are returned with an error and not charged. |
| `query` | string | No | Keyword to search the web for. The top results are fetched and converted to Markdown. |
| `maxResults` | integer | No | How many search results to scrape. Defaults to 10; at most 20. |

## Example

`?wait=true` holds the request open until the run finishes (up to 60s) and returns the rows inline: one request, no polling.

```bash
curl -X POST "https://api.mindcase.co/v1/data/web/scrape/run?wait=true" \
  -H "Authorization: Bearer mk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"params":{}}'
```

```python
import requests

resp = requests.post(
    "https://api.mindcase.co/v1/data/web/scrape/run",
    params={"wait": "true"},
    headers={"Authorization": "Bearer mk_live_YOUR_API_KEY"},
    json={"params": {}},
)
rows = resp.json()["data"]
```

## Longer runs

If the run is still going at the 60s ceiling you get back `{ job_id, status: "running" }` instead of rows (same if the result is over 100 rows; those come back `truncated`). Poll with that `job_id`:

```bash
# 1) check status
curl https://api.mindcase.co/v1/jobs/JOB_ID \
  -H "Authorization: Bearer mk_live_YOUR_API_KEY"

# 2) when status == completed, fetch the rows
curl https://api.mindcase.co/v1/jobs/JOB_ID/results \
  -H "Authorization: Bearer mk_live_YOUR_API_KEY"
```

Full API reference (auth, jobs, balance, SDKs, MCP): https://mindcase.co/skills.md

## Response columns

| Field | Display name | Type |
|-------|--------------|------|
| `url` | URL | text |
| `query` | Query | text |
| `rank` | Rank | number |
| `finalUrl` | Final URL | text |
| `statusCode` | Status Code | integer |
| `fetchStatus` | Fetch Status | text |
| `title` | Title | text |
| `domain` | Domain | text |
| `description` | Description | text |
| `snippet` | Snippet | text |
| `author` | Author | text |
| `publishedDate` | Published Date | text |
| `language` | Language | text |
| `wordCount` | Word Count | number |
| `markdown` | Markdown | text |
| `html` | Cleaned HTML | text |
| `htmlLength` | Html Length | integer |
| `meta` | Meta | object |
| `fetchedAt` | Fetched At | text |
| `engine` | Engine | text |
| `errorType` | Error Type | text |
| `errorMessage` | Error Message | text |