# crawler-with-api-and-mcp

> crawler-with-api-and-mcp — jawa7-crawler-with-api-and-mcp. Use this tool when you need to build a scalable web crawler with a modular architecture and API integration, to solve problems like data extraction and processing from multiple web sources. It takes in URLs and MCP protocols as inputs and outputs structured data, with automatic Swagger documentation for easy API management. Ideal for use cases requiring flexible and efficient web data extraction, with support for Git version control.

Canonical page: https://skillsregistry.net/skills/jawa7-crawler-with-api-and-mcp  
JSON: https://api.skillsregistry.net/v1/skills/jawa7-crawler-with-api-and-mcp

## Description

A learning project - a modular web crawler and API built with NestJS, supporting MCP (Model Context Protocol), TypeORM/PostgreSQL, and automatic Swagger documentation

## Trust

- **Trust score (0–1):** 0.87
- **Verification tier:** verified
- **Last scanned:** 2026-09-28

## Facts

- **Version:** 1.0.0
- **Skill type:** atomic
- **Execution layer:** container
- **Runtime environment:** vm
- **Updated:** 2026-09-28

## Source

- **Source listing:** [GitHub](https://github.com/jawa7/crawler-with-api-and-mcp)

## Use it

Resolve this record through the SkillsRegistry MCP server (no auth, read-only):

```
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_skill",
    "arguments": {
      "slug": "jawa7-crawler-with-api-and-mcp"
    }
  }
}
```

REST: `GET https://api.skillsregistry.net/v1/skills/jawa7-crawler-with-api-and-mcp` · pull for local use: `GET https://api.skillsregistry.net/v1/skills/jawa7-crawler-with-api-and-mcp/pull`

---
SkillsRegistry indexes agent skills from public registries and GitHub. Skills we have analysed are scanned with Circle-IR and scored on six dimensions; each listing states its scan coverage. More: https://skillsregistry.net/llms.txt