# Crawl4AI RAG

> Use this tool when you need to integrate web crawling and semantic search capabilities into AI agents or coding assistants, enabling them to crawl websites, store content, and perform intelligent queries over the crawled data. It solves problems related to information retrieval and knowledge gathering, providing inputs such as website URLs and outputs like relevant search results and stored content. Ideal for use cases where AI agents require access to up-to-date web data, such as coding assistance or research tasks.

Canonical page: https://skillsregistry.net/skills/coleam00-crawl4ai-rag  
JSON: https://api.skillsregistry.net/v1/skills/coleam00-crawl4ai-rag

## Description

This MCP server integrates web crawling capabilities from Crawl4AI with RAG (Retrieval Augmented Generation) functionality, enabling AI agents and coding assistants to crawl websites, store content in Supabase vector databases, and perform semantic searches over the crawled data. Developed by Cole Medin, it provides four essential tools: single page crawling, intelligent recursive website crawling with automatic URL type detection, source retrieval, and vector-based RAG queries with optional source filtering. The implementation is designed to be incorporated into Archon as a knowledge engine for AI coding assistants, with future plans to support different embedding models and local deployment options using Ollama.

## Trust

- **Trust score (0–1):** 0.55
- **Verification tier:** scanned
- **Last scanned:** 2026-09-28

## Facts

- **Version:** 1.0.0
- **Skill type:** atomic
- **Execution layer:** mcp-remote
- **Runtime environment:** api
- **Category:** database
- **Updated:** 2026-09-28

## Source

- **Source listing:** [PulseMCP](https://www.pulsemcp.com/servers/coleam00-crawl4ai-rag)
- **Repository:** <https://github.com/coleam00/mcp-crawl4ai-rag>

## Use it

Resolve this record through the SkillsRegistry MCP server (no auth, read-only):

```
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_skill",
    "arguments": {
      "slug": "coleam00-crawl4ai-rag"
    }
  }
}
```

REST: `GET https://api.skillsregistry.net/v1/skills/coleam00-crawl4ai-rag` · pull for local use: `GET https://api.skillsregistry.net/v1/skills/coleam00-crawl4ai-rag/pull`

---
SkillsRegistry indexes agent skills from public registries and GitHub. Skills we have analysed are scanned with Circle-IR and scored on six dimensions; each listing states its scan coverage. More: https://skillsregistry.net/llms.txt