SurfSense DocsWeb Crawl
Scrape any page or crawl a whole site into clean markdown
The web crawler fetches one or more pages — or spiders a whole site — and returns each page as clean markdown, plus metadata, every link with its anchor text, and harvested contact signals (emails, phone numbers, social profiles). JS-rendered pages are loaded in a real browser and auto-scrolled, so lazy-loaded listings are captured too.
Endpoint
POST /api/v1/workspaces/{workspace_id}/scrapers/web/crawlInputs
| Field | Default | Description |
|---|---|---|
startUrls | required | 1–20 seed URLs |
maxCrawlDepth | 0 | Link-hops to follow from each seed. 0 = fetch only the seeds; 1 = also the pages they link to; up to 5. The spider stays on the seed's site |
maxCrawlPages | 10 | Total pages fetched per call, seeds included (max 200) |
maxLength | 50000 | Max characters of markdown kept per page |
includeUrlPatterns | — | Regexes a discovered link must match to be followed (empty = follow every same-site link) |
excludeUrlPatterns | — | Regexes that exclude links from being followed (wins over include) |
Example
Scrape a single page:
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/web/crawl" \
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "startUrls": ["https://example.com/pricing"] }'Crawl a site's blog, two hops deep:
curl -X POST "$BASE_URL/api/v1/workspaces/1/scrapers/web/crawl" \
-H "Authorization: Bearer $SURFSENSE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"startUrls": ["https://example.com/blog"],
"maxCrawlDepth": 2,
"maxCrawlPages": 50,
"includeUrlPatterns": ["/blog/"]
}'Output
One item per fetched page (in crawl order) with its markdown, metadata (title, description), crawl provenance (depth, referrer), links classified as internal/external/social/email/tel, and page-level contacts. The response also includes a site-wide contacts summary that deduplicates every email, phone, and social profile found — siteWide: true marks header/footer values (the company's own contacts) versus page-local finds like individual team members.
Contact details often live on about/contact/privacy pages, so crawl with maxCrawlDepth >= 1 to surface them — useful for lead generation and competitive intelligence.
Billing is per successfully fetched page.
For the full input and output JSON schemas and generated code snippets in your language, open API Playground → Web → Crawl in your workspace.