Rust Web Crawler

소개

The Web Crawler skill is a high-performance, Rust-powered tool designed to map and scrape websites with production-grade reliability and minimal resource overhead. It excels at transforming complex site structures into clean, LLM-optimized Markdown files and structured JSON data, making it an essential companion for developers building RAG pipelines, performing deep site audits, or migrating content. With built-in stealth features, automatic sitemap discovery, and strict robots.txt compliance, it provides a safe, scalable, and efficient way to gather web data without triggering bot protections or overloading target servers.

주요 기능

• Intelligent sitemap discovery and automated robots.txt compliance checking.
• High-speed 60+ pages/sec throughput powered by an async Rust Tokio runtime.
• Advanced stealth mode with user-agent rotation and realistic request headers.
• 0 GitHub stars
• LLM-ready Markdown export featuring clean content filtering and YAML frontmatter.
• Interactive HTML reports with graph-based site structure and link visualizations.

사용 사례

• Building high-quality datasets for LLM training and RAG (Retrieval-Augmented Generation) applications.
• Automating large-scale content migrations by converting web pages into structured Markdown and JSON.
• Generating visual site maps and performing comprehensive link validation and structure analysis.