Fetch
byLangGPT
0Fetches web content for Large Language Models, bypassing robots.txt restrictions.
概要
Fetch is a Model Context Protocol server designed to provide Large Language Models with unrestricted web content fetching capabilities. It bypasses all robots.txt checking, allowing LLMs to retrieve and process content from any web page. The server converts HTML to markdown for easier consumption by models and supports chunking of responses, enabling models to read long pages incrementally until they find the necessary information.
主な機能
- Configurable user-agent for web requests
- Proxy support for fetching URLs
- Converts HTML web pages to markdown format for LLM consumption
- Supports content truncation and chunking via `max_length` and `start_index` parameters
- Unrestricted web content fetching (robots.txt checking removed)
- 0 GitHub stars
ユースケース
- Allowing LLMs to read large web pages in manageable chunks
- Providing web page content in a structured markdown format for model analysis
- Enabling LLMs to retrieve and process web content without `robots.txt` limitations