Read Website Fast
Extracts web content quickly and converts it to clean, token-efficient Markdown for AI agents and LLM pipelines.
About
Addresses the limitations of traditional web crawlers by offering a fast and token-efficient solution for AI agents. This tool fetches web pages locally, strips noise, and converts content to clean Markdown while preserving links, ensuring minimal token footprint and efficient processing for LLM pipelines and IDEs like Claude Code.
Key Features
- Converts HTML to clean Markdown using Mozilla Readability and Turndown GFM
- Link preservation for knowledge graphs and optional chunking for downstream processing
- Fast, token-efficient content extraction optimized for AI agents
- Configurable concurrent fetching and depth crawling for comprehensive data collection
- Smart caching with SHA-256 hashed URLs and polite crawling with robots.txt support
- 1 GitHub stars
Use Cases
- Powering AI agents with structured and clean web content
- Integrating clean web data into LLM pipelines and IDEs for context and research
- Efficient web scraping for data collection and analysis
Sponsored