SurfSense DocsWeb Crawl
Read a web page, or crawl a whole site into clean text
Web Crawl reads the web as clean, readable text. It opens any public page, or follows a site's links to read many of its pages at once, so the agent can summarise, compare or answer questions from what it found.
Needs a SurfSense license. See SurfSense plugins to connect it.
Tools
| Tool | What it does |
|---|---|
crawl | Reads one or more pages as text. It can also follow links from those pages, staying on the same site, a few levels deep. Each page comes back with its title, description, author, date and text, plus its links and any contact details it lists. |
The agent can choose how many link hops to follow, how many pages to read in all, and which links to follow or skip, for example only pages under /docs/.
Example prompts
- "@web-crawl Summarise https://example.com/blog/launch-post in five bullet points."
- "Read the pricing pages of these three products and compare their plans in a table: …"
- "Crawl the docs at https://docs.example.com, two levels deep, and tell me how authentication works."
- "Find the contact email and phone number on https://example.com."
Add from the web
Any link no other SurfSense plugin handles is read by Web Crawl as one page. Paste it in + › Add from the web… and click Add: the page's text becomes a source. Web Crawl doesn't open links as lists.
Limits
- Public pages only. Pages behind a login or a paywall stay out of reach.
- No local addresses. It refuses addresses on your own network, such as
localhostor192.168.…, and names that don't resolve. - Crawls have a ceiling. A crawl stops at five link hops and 200 pages. A very large crawl can time out, so ask for fewer pages if it does.
- Only http and https addresses can be read.
Last updated on November 2, 2026