# crawl-census

> crawl-census — taylorsmithgg-crawl-census-client. Use this tool when you need to efficiently crawl websites and avoid unnecessary requests, as it provides preflight checks and domain intelligence to identify domains that may refuse access or require payment. The crawl-census tool analyzes robots.txt files, edge refusals, and HTTP 402 paywalls to determine potential roadblocks. By using crawl-census, agents can optimize their crawling efforts and reduce wasted resources on inaccessible domains.

Canonical page: https://skillsregistry.net/skills/taylorsmithgg-crawl-census-client  
JSON: https://api.skillsregistry.net/v1/skills/taylorsmithgg-crawl-census-client

## Description

Provides preflight checks and domain intelligence so agents can skip domains that will refuse them or charge a fee, covering robots.txt, edge refusals, and HTTP 402 paywalls.

## Trust

- **Trust score (0–1):** 0.68
- **Verification tier:** scanned
- **Last scanned:** 2026-09-04

## Facts

- **Version:** 1.0.0
- **Skill type:** atomic
- **Execution layer:** mcp-remote
- **Runtime environment:** api
- **Category:** browser-automation
- **Updated:** 2026-09-04

## Source

- **Source listing:** [Glama](https://glama.ai/mcp/servers/iy0zyrn645)
- **Repository:** <https://github.com/taylorsmithgg/crawl-census-client>

## Use it

Resolve this record through the SkillsRegistry MCP server (no auth, read-only):

```
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_skill",
    "arguments": {
      "slug": "taylorsmithgg-crawl-census-client"
    }
  }
}
```

REST: `GET https://api.skillsregistry.net/v1/skills/taylorsmithgg-crawl-census-client` · pull for local use: `GET https://api.skillsregistry.net/v1/skills/taylorsmithgg-crawl-census-client/pull`

---
SkillsRegistry indexes agent skills from public registries and GitHub. Skills we have analysed are scanned with Circle-IR and scored on six dimensions; each listing states its scan coverage. More: https://skillsregistry.net/llms.txt