About ai.robots.txt
ai.robots.txt is a community-driven open-source list of user-agents for AI agents, scrapers, and web crawlers. The project provides pre-configured templates for robots.txt, .htaccess, Nginx, Caddy, HAProxy, and Traefik to help website owners easily restrict or block AI data harvesting.
Ideal for
Key Features
- Provides a comprehensive, community-maintained list of AI web crawler user-agents
- Includes ready-to-use configuration templates for Nginx, .htaccess, HAProxy, Caddy, and Traefik
- Fully open-source and free to implement under the permissive MIT license
- Supports automated updates through structured JSON configurations and GitHub Actions
- Helps website owners protect their content from unauthorized AI model training scraping
- Requires manual web server configuration or plugin setup to implement the blocks
- Relies on crawlers respecting robots.txt directives or using consistent user-agent headers
Alternatives to ai.robots.txt
Chatterbox
Open-Source Text-to-Speech Models
Unsloth
High-Performance Model Training and Fine-Tuning Library
Mastra
TypeScript AI Agent Framework
AI Website Cloner Template
AI Website Cloning Template
Gitagent
Git-Native Autonomous AI Agent Framework
Altic MCP
macOS Automation MCP Server for Claude
More Open Source Tools
Yargฤฑ MCP
MCP Server For Turkish Legal Databases
NVIDIA NemoClaw
Reference Stack For Sandboxed AI Agents
Temporal
Durable Execution and Workflow Orchestration Platform
Bento
AI Inference Platform
VoltAgent
TypeScript AI Agent Framework and Observability Platform
Cossistant
AI Support Framework For React And Next.js
More Libraries Tools
ArrowJS
Reactive UI Framework with WASM Sandboxing
tambo-ai
React AI Components
Reali3 API
Photogrammetry API for 3D Reconstruction
AgentMail
Email Inbox API for AI Agents
Descope
Customer and Agentic Identity Management Platform
Better i18n
AI-Powered Localization Infrastructure
