About ai.robots.txt
ai.robots.txt is a community-driven open-source list of user-agents for AI agents, scrapers, and web crawlers. The project provides pre-configured templates for robots.txt, .htaccess, Nginx, Caddy, HAProxy, and Traefik to help website owners easily restrict or block AI data harvesting.
Ideal for
Key Features
- Provides a comprehensive, community-maintained list of AI web crawler user-agents
- Includes ready-to-use configuration templates for Nginx, .htaccess, HAProxy, Caddy, and Traefik
- Fully open-source and free to implement under the permissive MIT license
- Supports automated updates through structured JSON configurations and GitHub Actions
- Helps website owners protect their content from unauthorized AI model training scraping
- Requires manual web server configuration or plugin setup to implement the blocks
- Relies on crawlers respecting robots.txt directives or using consistent user-agent headers
Alternatives to ai.robots.txt
Chatterbox
Open-Source Text-to-Speech Models
Unsloth
High-Performance Model Training and Fine-Tuning Library
Mastra
TypeScript AI Agent Framework
AI Website Cloner Template
AI Website Cloning Template
Gitagent
Git-Native Autonomous AI Agent Framework
Altic MCP
macOS Automation MCP Server for Claude
More Open Source Tools
Bento
AI Inference Platform
Cossistant
AI Support Framework For React And Next.js
VoltAgent
TypeScript AI Agent Framework and Observability Platform
Temporal
Durable Execution and Workflow Orchestration Platform
Microsoft VibeVoice
Generate Long-Form Multi-Speaker Conversational Audio
Selene
Local AI Assistant
