01Vector Search: Performs RAG over crawled content with optional source filtering.
02Content Chunking: Intelligently splits content by headers and size for better processing.
03Smart URL Detection: Automatically detects and handles different URL types (webpages, sitemaps, text files).
04Parallel Processing: Efficiently crawls multiple pages simultaneously.
05Recursive Crawling: Follows internal links to discover content.
06451 GitHub stars