# Web Crawler

> Use this tool when you need to extract content from websites in markdown format for AI-powered analysis. It solves problems of web content integration, allowing AI assistants to access and process online data without leaving the conversation interface. The Web Crawler tool takes website URLs as input and outputs extracted content in markdown format, making it ideal for workflows requiring web content analysis.

Canonical page: https://skillsregistry.net/skills/web-crawler  
JSON: https://api.skillsregistry.net/v1/skills/web-crawler

## Description

This MCP service provides a web crawling tool for Cursor and Claude applications, allowing AI assistants to extract content from websites in markdown format. Built using the FastMCP framework and Crawl4AI library, it's containerized with Docker for easy deployment on Render.com and includes a health check endpoint for monitoring. The implementation is designed to be easily connected to Cursor or Claude through their respective settings interfaces, making it ideal for workflows requiring web content analysis without leaving the conversation interface.

## Trust

- **Trust score (0–1):** 1.00
- **Verification tier:** verified
- **Last scanned:** 2026-09-19

## Facts

- **Version:** 1.0.0
- **Skill type:** atomic
- **Execution layer:** mcp-remote
- **Runtime environment:** api
- **Category:** cloud-infra
- **Updated:** 2026-09-19

## Source

- **Source listing:** [PulseMCP](https://www.pulsemcp.com/servers/web-crawler)
- **Repository:** <https://github.com/orange-fruit01/mcp-test-run>

## Use it

Resolve this record through the SkillsRegistry MCP server (no auth, read-only):

```
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_skill",
    "arguments": {
      "slug": "web-crawler"
    }
  }
}
```

REST: `GET https://api.skillsregistry.net/v1/skills/web-crawler` · pull for local use: `GET https://api.skillsregistry.net/v1/skills/web-crawler/pull`

---
SkillsRegistry indexes agent skills from public registries and GitHub. Skills we have analysed are scanned with Circle-IR and scored on six dimensions; each listing states its scan coverage. More: https://skillsregistry.net/llms.txt