About ai.robots.txt
ai.robots.txt is a community-driven open-source list of user-agents for AI agents, scrapers, and web crawlers. The project provides pre-configured templates for robots.txt, .htaccess, Nginx, Caddy, HAProxy, and Traefik to help website owners easily restrict or block AI data harvesting.
Ideal for
Key Features
- Provides a comprehensive, community-maintained list of AI web crawler user-agents
- Includes ready-to-use configuration templates for Nginx, .htaccess, HAProxy, Caddy, and Traefik
- Fully open-source and free to implement under the permissive MIT license
- Supports automated updates through structured JSON configurations and GitHub Actions
- Helps website owners protect their content from unauthorized AI model training scraping
- Requires manual web server configuration or plugin setup to implement the blocks
- Relies on crawlers respecting robots.txt directives or using consistent user-agent headers
Alternatives to ai.robots.txt
Chatterbox
Open-Source Text-to-Speech Models
Unsloth
High-Performance Model Training and Fine-Tuning Library
Mastra
TypeScript AI Agent Framework
AI Website Cloner Template
AI Website Cloning Template
Gitagent
Git-Native Autonomous AI Agent Framework
Altic MCP
macOS Automation MCP Server for Claude
More Open Source Tools
Yargฤฑ MCP
MCP Server For Turkish Legal Databases
NVIDIA NemoClaw
Reference Stack For Sandboxed AI Agents
VoltAgent
TypeScript AI Agent Framework and Observability Platform
Bento
AI Inference Platform
Cossistant
AI Support Framework For React And Next.js
Temporal
Durable Execution and Workflow Orchestration Platform
More Libraries Tools
Mem0
Self-Improving AI Memory Infrastructure Layer
tambo-ai
React AI Components
ArrowJS
Reactive UI Framework with WASM Sandboxing
Hugging Face
Open Source AI Platform
prompts.chat
Social Platform for AI Prompts
Browserbase
Headless Browser Platform for AI Agents
