Webustler

关于

Webustler is a powerful and free self-hosted Model Context Protocol (MCP) server designed for robust web scraping. Unlike most scraping tools that fail on protected sites, Webustler automatically bypasses Cloudflare and other anti-bot measures to deliver clean, token-optimized markdown. It extracts rich metadata, categorizes links, detects file downloads, and includes content analysis, making it ideal for feeding high-quality, structured data directly to large language models without the need for API keys or per-request charges.

主要功能

• Automatic anti-bot bypass, including Cloudflare protection
• Provides rich metadata: title, description, author, Open Graph, Twitter Cards, published time, etc.
• Categorizes internal and external links for structured crawling
• Smart fallback system with multiple retries for reliability
• 0 GitHub stars
• Extracts clean, LLM-ready markdown with tables preserved

使用案例

• Collecting content from protected sites without manual intervention
• Scraping articles and web content for LLM summarization
• Extracting structured data from websites for AI training