Web Scraper Toolkit

소개

Web Scraper Toolkit is a production-grade, multimodal scraping engine specifically engineered for AI Agents. It addresses the need for token-efficient, self-rectifying, and structured data by converting web content into easily consumable formats for Large Language Models. With its robust anti-bot evasion capabilities, including smart failover and stealth mode, and an explicit Model Context Protocol (MCP) server, it seamlessly integrates with agentic IDEs like Claude Desktop, providing a true sandbox environment for complex scraping workflows.

주요 기능

• Generates LLM-ready assets in Markdown, JSON, and PDF formats with a 'JSON Envelope' standard.
• Robust anti-bot evasion with intelligent Cloudflare/403 bypass and playwright-stealth.
• MCP native server for seamless integration with AI agents and agentic IDEs.
• Intelligent sitemap discovery with summary-first approach and keyword deep-search.
• Autonomous contact extraction and execution of complex JSON playbooks for multi-step missions.
• 0 GitHub stars

사용 사례

• Automating data collection for business intelligence, lead generation, and competitive analysis.
• Providing structured, token-efficient web data to AI agents and LLMs for research and content generation.
• Building robust data pipelines with resilient web scraping for analytics and reporting.