# Markdown Crawler

> Use this tool when you need to extract and organize markdown files from websites that follow Docsify-first crawling patterns. It solves problems of manually searching for and downloading markdown sources, and supports both stdio and HTTP deployment modes with secure bearer token authentication. The Markdown Crawler takes website URLs as input and outputs recursively fetched markdown files, preserving the original directory structure.

Canonical page: https://skillsregistry.net/skills/terryhank-markdown-crawler  
JSON: https://api.skillsregistry.net/v1/skills/terryhank-markdown-crawler

## Description

Identifies sites that follow Docsify-first crawling patterns and recursively fetches their publicly exposed markdown source files, preserving directory structure. Supports stdio and HTTP deployment modes with bearer token authentication.

## Trust

- **Trust score (0–1):** 0.94
- **Verification tier:** verified
- **Last scanned:** 2026-09-28

## Facts

- **Version:** 1.0.0
- **Skill type:** atomic
- **Execution layer:** mcp-remote
- **Runtime environment:** api
- **Category:** devops-ci
- **Updated:** 2026-09-28

## Source

- **Source listing:** [PulseMCP](https://www.pulsemcp.com/servers/terryhank-markdown-crawler)
- **Repository:** <https://github.com/terryhank/md_crawl_mcp>

## Use it

Resolve this record through the SkillsRegistry MCP server (no auth, read-only):

```
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_skill",
    "arguments": {
      "slug": "terryhank-markdown-crawler"
    }
  }
}
```

REST: `GET https://api.skillsregistry.net/v1/skills/terryhank-markdown-crawler` · pull for local use: `GET https://api.skillsregistry.net/v1/skills/terryhank-markdown-crawler/pull`

---
SkillsRegistry indexes agent skills from public registries and GitHub. Skills we have analysed are scanned with Circle-IR and scored on six dimensions; each listing states its scan coverage. More: https://skillsregistry.net/llms.txt