Web Crawler
This web crawler MCP server, implemented in TypeScript, provides a flexible and configurable tool for crawling websites and extracting content. It integrates with the Model Context Protocol SDK and uses libraries like Axios and Cheerio for efficient web scraping. The crawler respects robots.txt rules, handles concurrent requests, and offers customizable depth, delay, and timeout settings. It stands out by providing a simple JSON interface for initiating crawls and retrieving structured results, making it ideal for AI systems or applications that need to gather web content programmatically. Use cases include data collection for training models, content aggregation, or web monitoring tasks.
Composite of vulnerability cleanliness, spec conformance, provenance, stability, and usage signals — scanned and weighted by Cognium. Human and agent signals are tracked separately. Last scanned 2026-09-02.
Scan details: Circle-IR · 2026-09-02 · Appeal
View full trust & usage report →Metadata
- Version
- 1.0.0
- Skill type
- atomic
- Execution layer
- mcp-remote
- Category
- browser-automation
- Source
- PulseMCP
- Repository
- github.com/jitsmaster/webscrapemcpserver
- Author type
- human
- Last scanned
- 2026-09-02
- Updated
- 2026-09-02
Use via MCP
Resolve Web Crawler from your agent
Streamable HTTP transport at https://api.skillsregistry.net/mcp. No auth for read tools. Discovery: .well-known/mcp.json.
One command in your shell — Claude Code wires it up and verifies the connection. Run /mcp in any session to confirm.
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp--scope user for --scope project to commit it to .mcp.json.