Fetches web content for Large Language Models, bypassing robots.txt restrictions.
Fetch is a Model Context Protocol server designed to provide Large Language Models with unrestricted web content fetching capabilities. It bypasses all robots.txt checking, allowing LLMs to retrieve and process content from any web page. The server converts HTML to markdown for easier consumption by models and supports chunking of responses, enabling models to read long pages incrementally until they find the necessary information.
Key Features
01Configurable user-agent for web requests
02Proxy support for fetching URLs
03Converts HTML web pages to markdown format for LLM consumption
04Supports content truncation and chunking via `max_length` and `start_index` parameters
05Unrestricted web content fetching (robots.txt checking removed)
060 GitHub stars
Use Cases
01Allowing LLMs to read large web pages in manageable chunks
02Providing web page content in a structured markdown format for model analysis
03Enabling LLMs to retrieve and process web content without `robots.txt` limitations
