# Web Crawler

> Use this tool when you need to extract website content programmatically, and want a flexible and configurable solution for data collection, content aggregation, or web monitoring tasks. It takes in JSON inputs for crawl settings and outputs structured results, making it ideal for AI systems or applications. The web crawler is particularly useful for gathering data to train models, handling concurrent requests, and respecting website rules.

Canonical page: https://skillsregistry.net/skills/jitsmaster-web-crawler  
JSON: https://api.skillsregistry.net/v1/skills/jitsmaster-web-crawler

## Description

This web crawler MCP server, implemented in TypeScript, provides a flexible and configurable tool for crawling websites and extracting content. It integrates with the Model Context Protocol SDK and uses libraries like Axios and Cheerio for efficient web scraping. The crawler respects robots.txt rules, handles concurrent requests, and offers customizable depth, delay, and timeout settings. It stands out by providing a simple JSON interface for initiating crawls and retrieving structured results, making it ideal for AI systems or applications that need to gather web content programmatically. Use cases include data collection for training models, content aggregation, or web monitoring tasks.

## Trust

- **Trust score (0–1):** 0.65
- **Verification tier:** scanned
- **Last scanned:** 2026-09-02

## Facts

- **Version:** 1.0.0
- **Skill type:** atomic
- **Execution layer:** mcp-remote
- **Runtime environment:** api
- **Category:** browser-automation
- **Updated:** 2026-09-02

## Source

- **Source listing:** [PulseMCP](https://www.pulsemcp.com/servers/jitsmaster-web-crawler)
- **Repository:** <https://github.com/jitsmaster/webscrapemcpserver>

## Use it

Resolve this record through the SkillsRegistry MCP server (no auth, read-only):

```
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_skill",
    "arguments": {
      "slug": "jitsmaster-web-crawler"
    }
  }
}
```

REST: `GET https://api.skillsregistry.net/v1/skills/jitsmaster-web-crawler` · pull for local use: `GET https://api.skillsregistry.net/v1/skills/jitsmaster-web-crawler/pull`

---
SkillsRegistry indexes agent skills from public registries and GitHub. Skills we have analysed are scanned with Circle-IR and scored on six dimensions; each listing states its scan coverage. More: https://skillsregistry.net/llms.txt