# Web Crawl API

Crawl a website from one start URL and get every page it links to as clean Markdown, with title, description and link depth. 50 pages by default, up to 1,000. Only pages that came back are charged.

**Pricing:** $0.003 per page

**Endpoint:** `POST /v1/data/web/crawl/run`

**Auth:** `Authorization: Bearer mk_live_...`

## Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `url` | string | Yes | The page to start from, like https://docs.example.com. Links are followed from here. |
| `maxPages` | integer | No | How many pages to crawl. Defaults to 50; at most 1,000. |
| `includePaths` | array | No | Only crawl paths that match, like /blog/* or /docs/*. |
| `excludePaths` | array | No | Skip paths that match, like /tag/* or /author/*. |

## Example

`?wait=true` holds the request open until the run finishes (up to 60s) and returns the rows inline: one request, no polling.

```bash
curl -X POST "https://api.mindcase.co/v1/data/web/crawl/run?wait=true" \
  -H "Authorization: Bearer mk_live_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"params":{"url":"..."}}'
```

```python
import requests

resp = requests.post(
    "https://api.mindcase.co/v1/data/web/crawl/run",
    params={"wait": "true"},
    headers={"Authorization": "Bearer mk_live_YOUR_API_KEY"},
    json={"params": {
    "url": "..."
}},
)
rows = resp.json()["data"]
```

## Longer runs

If the run is still going at the 60s ceiling you get back `{ job_id, status: "running" }` instead of rows (same if the result is over 100 rows; those come back `truncated`). Poll with that `job_id`:

```bash
# 1) check status
curl https://api.mindcase.co/v1/jobs/JOB_ID \
  -H "Authorization: Bearer mk_live_YOUR_API_KEY"

# 2) when status == completed, fetch the rows
curl https://api.mindcase.co/v1/jobs/JOB_ID/results \
  -H "Authorization: Bearer mk_live_YOUR_API_KEY"
```

Full API reference (auth, jobs, balance, SDKs, MCP): https://mindcase.co/skills.md

## Response columns

| Field | Display name | Type |
|-------|--------------|------|
| `url` | URL | text |
| `title` | Title | text |
| `description` | Description | text |
| `markdown` | Markdown | text |
| `wordCount` | Word Count | number |
| `depth` | Depth | integer |
| `statusCode` | Status Code | integer |
| `fetchedAt` | Fetched At | text |
| `engine` | Engine | text |
| `servedFromCache` | Served From Cache | boolean |