Fetch
Fetches web content for Large Language Models, bypassing robots.txt restrictions.
About
Fetch is a Model Context Protocol server designed to provide Large Language Models with unrestricted web content fetching capabilities. It bypasses all robots.txt checking, allowing LLMs to retrieve and process content from any web page. The server converts HTML to markdown for easier consumption by models and supports chunking of responses, enabling models to read long pages incrementally until they find the necessary information.
Key Features
- Configurable user-agent for web requests
- Proxy support for fetching URLs
- Converts HTML web pages to markdown format for LLM consumption
- Supports content truncation and chunking via `max_length` and `start_index` parameters
- Unrestricted web content fetching (robots.txt checking removed)
- 0 GitHub stars
Use Cases
- Allowing LLMs to read large web pages in manageable chunks
- Providing web page content in a structured markdown format for model analysis
- Enabling LLMs to retrieve and process web content without `robots.txt` limitations
Sponsored