About
This open-source project provides a production-style, horizontally scalable distributed search engine. It features a robust Python-based crawler and indexing pipeline, leveraging Postgres, batch jobs, and a FastAPI for serving search results for both web and news. Designed for efficiency, it incorporates fast stateless workers for high crawl throughput, offline global computation for metrics like PageRank and BM25, and a migration-first schema. Deployment is streamlined with Docker Compose and Swarm compatibility, enabling developers to quickly establish a powerful, self-hosted search solution, complete with an MCP server for integrated search tools.
