# web-crawler-mcp-server

> web-crawler-mcp-server — insightful-pipe-web-crawler-mcp-server. Use this tool when you need to crawl and scrape websites for SEO audits, as it provides a remote MCP server to extract relevant data. It solves problems related to website analysis and optimization by integrating with various AI models like Claude, ChatGPT, Cursor, and Claude Code. It takes website URLs as input and outputs scraped data for further analysis, ideal for use cases requiring automated website audits and SEO monitoring.

Canonical page: https://skillsregistry.net/skills/insightful-pipe-web-crawler-mcp-server  
JSON: https://api.skillsregistry.net/v1/skills/insightful-pipe-web-crawler-mcp-server

## Description

Web crawler MCP server: crawl and scrape websites for SEO audits from Claude, ChatGPT, Cursor and Claude Code. Hosted remote MCP.

## Trust

- **Trust score (0–1):** 1.00
- **Verification tier:** verified
- **Last scanned:** 2026-09-28

## Facts

- **Version:** 1.0.0
- **Skill type:** atomic
- **Execution layer:** container
- **Runtime environment:** vm
- **License:** MIT
- **Updated:** 2026-09-28

## Source

- **Source listing:** [GitHub](https://github.com/Insightful-Pipe/web-crawler-mcp-server)

## Use it

Resolve this record through the SkillsRegistry MCP server (no auth, read-only):

```
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_skill",
    "arguments": {
      "slug": "insightful-pipe-web-crawler-mcp-server"
    }
  }
}
```

REST: `GET https://api.skillsregistry.net/v1/skills/insightful-pipe-web-crawler-mcp-server` · pull for local use: `GET https://api.skillsregistry.net/v1/skills/insightful-pipe-web-crawler-mcp-server/pull`

---
SkillsRegistry indexes agent skills from public registries and GitHub. Skills we have analysed are scanned with Circle-IR and scored on six dimensions; each listing states its scan coverage. More: https://skillsregistry.net/llms.txt