Tomoshibi

•
4
•

Provides a self-hosted server for web scraping and search, transforming any website content into LLM-ready markdown quickly and efficiently.

Tomoshibi is a high-performance, self-hosted search and scraping server designed specifically for AI agents. Built with Go, it rapidly converts web pages into clean, LLM-ready markdown, often in as little as 200ms. It addresses common pain points of hosted scraping APIs by offering a cost-effective, single-binary solution that runs efficiently on minimal resources (e.g., $5/month hosting). Featuring a 'smart mode' for handling both static and dynamic JavaScript-heavy sites, it ensures accurate content extraction while stripping away noise like ads and navigation. Tomoshibi also boasts a monolithic architecture for simplified deployment, offering API compatibility with Firecrawl's schema, making it an ideal choice for building Retrieval Augmented Generation (RAG) systems without incurring high cloud bills.

Key Features

014 GitHub stars
02Self-hosted web scraping and search API at minimal cost
03Rapid page conversion to LLM-ready markdown (avg. 200ms)
04Smart mode for dynamic (JS-SPA) and static site scraping
05Clean, main-content extraction with ad-blocking for LLM consumption
06Monolithic architecture (Go API + Asynq) for simplified deployment

Use Cases

01Building cost-effective Retrieval Augmented Generation (RAG) systems
02Monitoring website content changes and triggering webhooks
03Collecting up-to-date web data for AI agents and large language models