# site-crawler-mcp

> site-crawler-mcp — coolaigit-site-crawler-mcp. Use this tool when you need to crawl and analyze website content, as it enables site-wide crawling of internal links with filtering and page operations, solving problems like data extraction and content summarization. It takes in website URLs and filtering parameters as inputs, and outputs summarized content and link data persisted to SQLite. Ideal for use cases requiring automated website data collection and analysis, such as research, monitoring, or archiving.

Canonical page: https://skillsregistry.net/skills/coolaigit-site-crawler-mcp  
JSON: https://api.skillsregistry.net/v1/skills/coolaigit-site-crawler-mcp

## Description

Enables site-wide crawling of internal links with BFS traversal, time/title/URL filtering, and page operations like summarization and link clicking, with results persisted to SQLite.

## Trust

- **Trust score (0–1):** 0.69
- **Verification tier:** scanned
- **Last scanned:** 2026-09-03

## Facts

- **Version:** 1.0.0
- **Skill type:** atomic
- **Execution layer:** mcp-remote
- **Runtime environment:** api
- **Category:** database
- **Updated:** 2026-09-03

## Source

- **Source listing:** [Glama](https://glama.ai/mcp/servers/zq6uuy2fn5)
- **Repository:** <https://github.com/coolaigit/site-crawler-mcp>

## Use it

Resolve this record through the SkillsRegistry MCP server (no auth, read-only):

```
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_skill",
    "arguments": {
      "slug": "coolaigit-site-crawler-mcp"
    }
  }
}
```

REST: `GET https://api.skillsregistry.net/v1/skills/coolaigit-site-crawler-mcp` · pull for local use: `GET https://api.skillsregistry.net/v1/skills/coolaigit-site-crawler-mcp/pull`

---
SkillsRegistry indexes agent skills from public registries and GitHub. Skills we have analysed are scanned with Circle-IR and scored on six dimensions; each listing states its scan coverage. More: https://skillsregistry.net/llms.txt