Crawls entire websites and builds searchable full-text indexes of content converted into Markdown format.
This skill enables Claude to deeply explore websites by crawling multiple pages, converting HTML content into clean Markdown, and building a local Tantivy-powered search index. It is specifically designed for technical research and documentation ingestion, allowing users to perform advanced full-text searches—including fuzzy matching and boolean logic—across an entire site to find specific implementation details or code patterns without manual browsing. All data is stored locally in your project directory, making it an ideal tool for creating offline knowledge bases or preparing data for RAG workflows.
主な機能
01Automated multi-level web crawling with customizable depth and rate limiting.
02Background execution capabilities with progress monitoring and parallel crawl management.
03Full-text search indexing using the Tantivy engine for high-performance local queries.
04Support for advanced search syntax including fuzzy matching, phrases, and boolean operators.
050 GitHub stars
06Automatic conversion of complex HTML into clean, readable Markdown files.
ユースケース
01Converting technical blogs or wikis into a searchable, local Markdown knowledge base.
02Indexing a complete software documentation site to quickly find API usage examples and edge cases.
03Scraping domain-specific content to provide context for complex coding tasks and architectural decisions.
