# Web Crawler MCP Server

> Use this tool when you need to extract relevant information from dynamic websites, as it solves the problem of accessing and filtering website content through natural language queries. The Web Crawler MCP Server takes in natural language queries as input and outputs relevant links, while providing secure access through GitHub OAuth authentication. It is ideal for use cases where AI assistants require web data extraction, such as data mining, research, and content aggregation.

Canonical page: https://skillsregistry.net/skills/dhannusch-smart-web-crawler-mcp  
JSON: https://api.skillsregistry.net/v1/skills/dhannusch-smart-web-crawler-mcp

## Description

An intelligent web crawling server that uses Cloudflare's headless browser to render dynamic pages and Workers AI to extract relevant links based on natural language queries. It enables AI assistants to search and filter website content while providing secure access through GitHub OAuth authentication.

## Trust

- **Trust score (0–1):** 0.69
- **Verification tier:** scanned
- **Last scanned:** 2026-09-02

## Facts

- **Version:** 1.0.0
- **Skill type:** atomic
- **Execution layer:** mcp-remote
- **Runtime environment:** api
- **Category:** version-control
- **Updated:** 2026-09-02

## Source

- **Source listing:** [Glama](https://glama.ai/mcp/servers/fd80goyf9k)
- **Repository:** <https://github.com/dhannusch/smart-web-crawler-mcp>

## Use it

Resolve this record through the SkillsRegistry MCP server (no auth, read-only):

```
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_skill",
    "arguments": {
      "slug": "dhannusch-smart-web-crawler-mcp"
    }
  }
}
```

REST: `GET https://api.skillsregistry.net/v1/skills/dhannusch-smart-web-crawler-mcp` · pull for local use: `GET https://api.skillsregistry.net/v1/skills/dhannusch-smart-web-crawler-mcp/pull`

---
SkillsRegistry indexes agent skills from public registries and GitHub. Skills we have analysed are scanned with Circle-IR and scored on six dimensions; each listing states its scan coverage. More: https://skillsregistry.net/llms.txt