WebShift
Transforms noisy web pages into clean, context-ready text optimized for Large Language Models.
WebShift is a Rust library and Model Context Protocol (MCP) server designed to process raw HTML from web pages, stripping away common clutter like scripts, ads, and navigation menus. It sterilizes text, manages output size with strict budgets, and provides a streamlined pipeline from search query to structured, clean content. This ensures that Large Language Models receive only relevant information, maximizing their context window efficiency and reasoning capabilities.
主な機能
013 GitHub stars
02Enforces strict size budgets and anti-flooding protections for context window efficiency.
03Applies deterministic BM25 and optional LLM-assisted reranking of search results.
04Integrates with 8 search backends for comprehensive web content retrieval.
05Offers optional LLM features like query expansion and Markdown summarization with citations.
06Strips noise from HTML pages to produce clean, LLM-ready text.
ユースケース
01Building AI agents that require clean, budget-controlled web search results.
02Integrating a robust HTML-to-text cleaner into Rust pipelines for web content processing.
03Deploying a single-binary MCP web search server without external dependencies.