# CrawMcp

> Use this tool when you need to bypass anti-bot walls and JS challenges for web crawling, enabling asynchronous content handling with automatic cleanup. It solves problems of accessing restricted web content and handles large volumes of data with RabbitMQ, MinIO, and Redis integration. Ideal for use cases requiring self-hosted, efficient, and scalable web crawling solutions.

Canonical page: https://skillsregistry.net/skills/bilginerdem-craw-mcp  
JSON: https://api.skillsregistry.net/v1/skills/bilginerdem-craw-mcp

## Description

Self-hosted MCP server for web crawling that bypasses anti-bot walls and JS challenges, using RabbitMQ, MinIO, and Redis for asynchronous content handling with automatic cleanup.

## Trust

- **Trust score (0–1):** 0.85
- **Verification tier:** verified
- **Last scanned:** 2026-09-19

## Facts

- **Version:** 1.0.0
- **Skill type:** atomic
- **Execution layer:** mcp-remote
- **Runtime environment:** api
- **Category:** database
- **Updated:** 2026-09-19

## Source

- **Source listing:** [Glama](https://glama.ai/mcp/servers/k46cug6v03)
- **Repository:** <https://github.com/bilginerdem/craw-mcp>

## Use it

Resolve this record through the SkillsRegistry MCP server (no auth, read-only):

```
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_skill",
    "arguments": {
      "slug": "bilginerdem-craw-mcp"
    }
  }
}
```

REST: `GET https://api.skillsregistry.net/v1/skills/bilginerdem-craw-mcp` · pull for local use: `GET https://api.skillsregistry.net/v1/skills/bilginerdem-craw-mcp/pull`

---
SkillsRegistry indexes agent skills from public registries and GitHub. Skills we have analysed are scanned with Circle-IR and scored on six dimensions; each listing states its scan coverage. More: https://skillsregistry.net/llms.txt