# Spider Web Scraper

> Use this tool when you need to scrape webpage content, perform web searches, or convert pages to Markdown format without relying on external search APIs. It solves problems related to competitive research, content analysis, and building AI assistants that require programmatic access to web data. The Spider Web Scraper takes in URLs and concurrency limits as inputs and outputs scraped content, search results, or Markdown-formatted pages.

Canonical page: https://skillsregistry.net/skills/spider-web-scraper  
JSON: https://api.skillsregistry.net/v1/skills/spider-web-scraper

## Description

Spider MCP is a web search and scraping server implementation by Bosegluon that provides pure web scraping capabilities without relying on external search APIs. It integrates Puppeteer for browser automation and Cheerio for HTML parsing to perform web searches across Bing (including news search with time filters), scrape webpage content, and convert pages to Markdown format. The server supports batch processing of multiple URLs with configurable concurrency limits and includes comprehensive error handling, making it useful for competitive research, content analysis, and building AI assistants that need programmatic access to web data without API dependencies or rate limits.

## Trust

- **Trust score (0–1):** 0.82
- **Verification tier:** verified
- **Last scanned:** 2026-09-28

## Facts

- **Version:** 1.0.0
- **Skill type:** atomic
- **Execution layer:** mcp-remote
- **Runtime environment:** api
- **Category:** browser-automation
- **Updated:** 2026-09-28

## Source

- **Source listing:** [PulseMCP](https://www.pulsemcp.com/servers/spider-web-scraper)
- **Repository:** <https://github.com/yc9yc/spider-mcp>

## Use it

Resolve this record through the SkillsRegistry MCP server (no auth, read-only):

```
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_skill",
    "arguments": {
      "slug": "spider-web-scraper"
    }
  }
}
```

REST: `GET https://api.skillsregistry.net/v1/skills/spider-web-scraper` · pull for local use: `GET https://api.skillsregistry.net/v1/skills/spider-web-scraper/pull`

---
SkillsRegistry indexes agent skills from public registries and GitHub. Skills we have analysed are scanned with Circle-IR and scored on six dimensions; each listing states its scan coverage. More: https://skillsregistry.net/llms.txt