# robotstxt-ai-mcp

> Use this tool when you need to manage and optimize your website's robots.txt file for AI crawler access, solving issues with bot traffic and crawl management. It analyzes existing files, detects blocked or allowed AI bots, and generates optimized versions with customizable controls for 20+ AI crawlers. Ideal for webmasters and SEO specialists seeking to refine their site's crawlability and indexing by various AI-powered bots.

Canonical page: https://skillsregistry.net/skills/sharozdawa-robotstxt-ai  
JSON: https://api.skillsregistry.net/v1/skills/sharozdawa-robotstxt-ai

## Description

Analyze and generate robots.txt files with AI crawler awareness. Fetch any site's robots.txt, detect which AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) are blocked or allowed, and generate optimized robots.txt with toggle controls for 20+ AI crawlers.

## Trust

- **Trust score (0–1):** 0.95
- **Verification tier:** verified
- **Last scanned:** 2026-09-19

## Facts

- **Version:** 1.0.0
- **Skill type:** atomic
- **Execution layer:** mcp-remote
- **Runtime environment:** api
- **Category:** browser-automation
- **Updated:** 2026-09-19

## Source

- **Source listing:** [Glama](https://glama.ai/mcp/servers/t1cy2qwu3y)
- **Repository:** <https://github.com/sharozdawa/robotstxt-ai>

## Use it

Resolve this record through the SkillsRegistry MCP server (no auth, read-only):

```
claude mcp add --transport http --scope user skillsregistry https://api.skillsregistry.net/mcp
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_skill",
    "arguments": {
      "slug": "sharozdawa-robotstxt-ai"
    }
  }
}
```

REST: `GET https://api.skillsregistry.net/v1/skills/sharozdawa-robotstxt-ai` · pull for local use: `GET https://api.skillsregistry.net/v1/skills/sharozdawa-robotstxt-ai/pull`

---
SkillsRegistry indexes agent skills from public registries and GitHub. Skills we have analysed are scanned with Circle-IR and scored on six dimensions; each listing states its scan coverage. More: https://skillsregistry.net/llms.txt