Webcrawl
Createdpragmar
Connects web crawler data and archives to AI language models via the Model Context Protocol (MCP).
About
Webcrawl bridges the gap between web crawl data and AI language models using the Model Context Protocol (MCP). It enables AI clients to filter and analyze web content under user direction or autonomously, offering a full-text search interface with boolean support and resource filtering. Compatible with various web crawlers like WARC, wget, InterroBot, Katana, and SiteOne, it provides LLMs with a comprehensive menu for searching web content.
Key Features
- Full-text search support
- Quick MCP configuration
- Multi-crawler compatible
- Claude Desktop ready
- Filter by type and HTTP status
- 1 GitHub stars
Use Cases
- Filtering and analyzing web content with AI clients
- Connecting web crawl data to LLMs
- Searching web archives using full-text search
