Docs FAQs

Question 1

What is Docs and what problem does it solve?

Accepted Answer

Docs is a specialized software tool designed to transform raw web pages into clean, structured context suitable for large language models (LLMs). It solves the challenge of obtaining up-to-date, relevant web content for applications like Retrieval-Augmented Generation (RAG).

Question 2

What type of data does Docs store and how is it managed?

Accepted Answer

Docs stores structured data like sources, pages, and crawl jobs in PostgreSQL. Crawled HTML content is persistently stored in S3, ensuring scalability and reliable access to the raw web data. Alembic is used for efficient database schema migrations.

Question 3

How does Docs transform web pages into LLM-ready context?

Accepted Answer

Docs utilizes asynchronous web crawling capabilities with Celery, httpx, and Playwright to efficiently scrape web content. The collected data is then processed and stored, making it readily available for LLM consumption and integration into various AI workflows.

Question 4

Can Docs be integrated into existing applications?

Accepted Answer

Yes, Docs provides a FastAPI-based API (`/api/v1`) for seamless integration. Its modular design ensures that various components can be consumed independently, allowing for flexible deployment and custom application development.

Question 5

What are the core architectural components of Docs?

Accepted Answer

Docs features a modular architecture with independent `backend` (FastAPI), `mcp_server`, and `crawler` components. These modules leverage PostgreSQL and S3 for persistent data storage, Redis for messaging, and Alembic for robust database schema management.

Docs

Docs

Key Features

Use Cases

Key Features

Use Cases