WebPeel

About

WebPeel is a powerful web fetching tool designed specifically for AI agents, capable of converting any web page into clean, AI-ready markdown. It features smart escalation, automatically progressing from fast HTTP fetches to browser rendering, and even a stealth mode to bypass bot detection when necessary. With support for both Node.js and Python SDKs, a global CLI, and a self-hostable Docker setup, WebPeel offers robust web crawling, mapping, and searching capabilities. It also integrates seamlessly with AI tools via its MCP server, providing a suite of tools for autonomous agents to perform web research using your own LLM key.

Key Features

• 0 GitHub stars
• Smart Escalation (HTTP → Browser → Stealth)
• Stealth Mode for bypassing bot detection
• Web Crawling and Mapping with sitemap and robots.txt compliance
• Autonomous Agent with Bring Your Own Key (BYOK) LLM support
• Comprehensive MCP Server integration with 9 tools for AI agents

Use Cases

• Fetching clean, AI-ready markdown content from web pages for large language models.
• Crawling and mapping websites to build knowledge bases or perform extensive data collection.
• Empowering autonomous AI agents to research the web and gather information efficiently.