Does Webshift offer integrated web search capabilities?

Yes, it includes a Web Content Client with streaming HTTP fetching, integration with 8 search backends (e.g., SearXNG, Google), BM25 reranking, and optional LLM-assisted query expansion.

What is Webshift primarily designed for?

Webshift is a Rust library and MCP server that cleans noisy web pages, extracting only relevant, right-sized text optimized for Large Language Model (LLM) consumption and AI agents.

How does Webshift improve LLM context window efficiency?

It strips irrelevant elements like scripts, ads, and navigation from raw HTML, sterilizes text, and enforces strict size budgets, ensuring LLMs receive only essential content, maximizing context usage.

What kind of web pages can Webshift process?

Webshift parses static HTML to extract text. It is not designed for rendering JavaScript-heavy Single Page Applications (SPAs) or complex web scraping tasks like form submission or authentication flows.

Webshift: Clean Web Pages for LLM Context & AI Agents

WebShift is a powerful Rust library and MCP server designed to transform chaotic web pages into pristine, context-ready text for LLMs. It meticulously strips away extraneous elements like scripts, ads, navigation menus, and cookie banners, preventing LLM context windows from being flooded with irrelevant data. By sterilizing text, enforcing strict size budgets, and offering advanced features like search integration with BM25 reranking and optional LLM-driven query expansion and summarization, WebShift ensures that models receive only the critical information they need for effective reasoning and response generation.

Características Principales

01MCP Server: Provides a native `mcp-webshift` binary that exposes web search, fetch, and onboarding functionalities over the Model Context Protocol.

021 GitHub stars

03HTML Denoiser: Pure Rust pipeline for stripping noise elements, sterilizing Unicode, and collapsing whitespace from raw HTML, optimized for LLM input.

04Web Content Client: Integrates streaming HTTP fetching with size caps, 8 search backends (e.g., SearXNG, Brave, Google), and BM25 reranking for comprehensive web search.

05LLM-Assisted Processing: Optional features include query expansion for generating multiple search variants, Markdown summarization with inline citations, and LLM-assisted tier-2 reranking.

06Anti-flooding Protections: Implements strict controls like streaming download limits, per-page character caps, total character budgets, and binary filtering to prevent information overload.

Casos de Uso

01Equipping AI agents with clean, budget-controlled web search capabilities for enhanced reasoning.

02Integrating a reliable HTML-to-text cleaner into Rust pipelines for processing web content without noise.

03Deploying a single-binary MCP web search server without complex runtime dependencies like Python or Docker.