About
Crawl4AI integrates web scraping and crawling as tools for Large Language Models, specifically within Cursor Composer's agent mode. It provides functionality for single page scraping with metadata extraction, as well as multi-page crawling with configurable depth and limits. The tool supports markdown and JSON output formats and offers a simple command-line interface for easy integration.
Key Features
- Single page scraping with metadata extraction
- Markdown and JSON output formats
- Simple command-line interface
- Easy integration with Cursor AI
- 1 GitHub stars
- Multi-page crawling with configurable depth and limits
Use Cases
- Provides web scraping capabilities for Cursor AI
- Extracts and formats web data for AI agents
- Crawls multiple webpages for LLM context
