searxNcrawl
Equip AI agents with web search capabilities and clean Markdown web crawling powered by Crawl4AI and SearXNG.
searxNcrawl is a self-hosted MCP server and CLI toolkit that bridges AI agents to private web search and high-quality web crawling without requiring external API keys. Built on Crawl4AI and Playwright Chromium, it can render JavaScript-heavy sites, deduplicate repeated page blocks, crawl full documentation structures, and even bypass authentication barriers via session capture. By querying private SearXNG instances or You.com in parallel and returning deduplicated Markdown or JSON, searxNcrawl offers a lightweight, privacy-focused alternative to hosted search and extraction services.
主な機能
01185 GitHub stars
02Multi-provider parallel web search supporting SearXNG and You.com with merged, attributed results
03Flexible deployment via local STDIO or HTTP transport with host and CORS guards
04Deep site crawling with configurable depth, page limits, and internal link traversal tailored for documentation
05Clean Markdown and JSON page conversion powered by Crawl4AI and Playwright Chromium
06Authenticated crawling support using Playwright session capture for pages behind logins
ユースケース
01Grounding LLM coding assistants and agents in up-to-date documentation using direct site crawling
02Providing private, self-hosted web search and page retrieval without recurring third-party API costs
03Scraping and summarizing internal or authenticated web portals for RAG pipelines