01FastAPI-based API for seamless integration and consumption
02Persistent data storage with PostgreSQL and S3 for crawled content
03Integrated Alembic for robust database schema management
04Modular architecture with independent backend, server, and crawler components
05Asynchronous web crawling capabilities using Celery, httpx, and Playwright
060 GitHub stars