# TheCrawler

> TheCrawler — manchittlab-thecrawler. Use this tool when you need to extract structured data from websites, documents, or web pages, and convert unstructured content into usable formats like JSON-LD or markdown. It solves problems of data scraping, extraction, and processing, handling various file types and detecting analytics trackers. TheCrawler accepts inputs via CLI, npm, REST API, or MCP server, and outputs structured data with error handling and retry capabilities.

Canonical page: https://skillsregistry.net/skills/manchittlab-thecrawler  
JSON: https://api.skillsregistry.net/v1/skills/manchittlab-thecrawler

## Description

Open-source web scraper + LLM-powered structured extraction. PDF/DOCX, markdown, JSON-LD, microdata, commerce data, forms, 16 analytics-tracker detection. Structured errors with retryable flags. Adaptive Cheerio->Playwright. CLI, npm, REST API, and MCP server. AGPL-3.0.

## Trust

- **Trust score (0–1):** 0.54
- **Verification tier:** scanned
- **Last scanned:** 2026-09-01

## Facts

- **Version:** 1.0.0
- **Skill type:** atomic
- **Execution layer:** container
- **Runtime environment:** vm
- **Category:** browser-automation
- **Updated:** 2026-09-21

## Source

- **Source listing:** [GitHub](https://github.com/manchittlab/TheCrawler)

## Use it

Resolve this record through the SkillsRegistry MCP server (no auth, read-only):

```
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_skill",
    "arguments": {
      "slug": "manchittlab-thecrawler"
    }
  }
}
```

REST: `GET https://api.skillsregistry.net/v1/skills/manchittlab-thecrawler` · pull for local use: `GET https://api.skillsregistry.net/v1/skills/manchittlab-thecrawler/pull`

---
SkillsRegistry indexes agent skills from public registries and GitHub. Skills we have analysed are scanned with Circle-IR and scored on six dimensions; each listing states its scan coverage. More: https://skillsregistry.net/llms.txt