# crawler-mcp

> Use this tool when you need to fetch and extract specific content from web pages, and automate site crawling with precision. The crawler-mcp server solves problems of dynamic web page content extraction and link discovery, accepting inputs such as URLs and CSS selectors, and outputting extracted data and links. It is ideal for use cases requiring automated web data extraction and site crawling, providing a flexible interface for custom content selection.

Canonical page: https://skillsregistry.net/skills/shadab15github-crawler-mcp  
JSON: https://api.skillsregistry.net/v1/skills/shadab15github-crawler-mcp

## Description

Web-crawler MCP server that fetches web pages with static parsing or headless browser fallback, enabling Claude to fetch, extract links, crawl sites, and select content via CSS selectors.

## Trust

- **Trust score (0–1):** 0.70
- **Verification tier:** verified
- **Last scanned:** 2026-09-03

## Facts

- **Version:** 1.0.0
- **Skill type:** atomic
- **Execution layer:** mcp-remote
- **Runtime environment:** api
- **Category:** browser-automation
- **Updated:** 2026-09-03

## Source

- **Source listing:** [Glama](https://glama.ai/mcp/servers/rbaqz137fb)
- **Repository:** <https://github.com/shadab15github/crawler-mcp>

## Use it

Resolve this record through the SkillsRegistry MCP server (no auth, read-only):

```
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_skill",
    "arguments": {
      "slug": "shadab15github-crawler-mcp"
    }
  }
}
```

REST: `GET https://api.skillsregistry.net/v1/skills/shadab15github-crawler-mcp` · pull for local use: `GET https://api.skillsregistry.net/v1/skills/shadab15github-crawler-mcp/pull`

---
SkillsRegistry indexes agent skills from public registries and GitHub. Skills we have analysed are scanned with Circle-IR and scored on six dimensions; each listing states its scan coverage. More: https://skillsregistry.net/llms.txt