Enables web scraping and crawling capabilities for Large Language Models.
Crawl4AI integrates web scraping and crawling as tools for Large Language Models, specifically within Cursor Composer's agent mode. It provides functionality for single page scraping with metadata extraction, as well as multi-page crawling with configurable depth and limits. The tool supports markdown and JSON output formats and offers a simple command-line interface for easy integration.
Key Features
01Single page scraping with metadata extraction
02Markdown and JSON output formats
03Simple command-line interface
04Easy integration with Cursor AI
051 GitHub stars
06Multi-page crawling with configurable depth and limits
Use Cases
01Provides web scraping capabilities for Cursor AI