How does this skill handle service failures?

It implements 'unless-stopped' Docker restart policies and systemd 'always' restart configurations to ensure services automatically recover from crashes or server reboots.

What does the Scraper Wrapper include?

The wrapper is a Bun-based service that handles the Firecrawl API requests, saves the resulting markdown to a local directory, and returns a public URL for easy access.

Firecrawl is an open-source engine that crawls websites and converts them into clean markdown format specifically optimized for LLM use cases.

Why use a self-hosted Firecrawl instance?

Self-hosting provides full control over data privacy, removes external API costs, and allows you to customize scraping depth and rate limits for your specific needs.

Firecrawl Self-Hosted Manager

Name: Firecrawl Self-Hosted Manager
Author: terrylica

byterrylica

•

Web Scraping y Recopilación de Datos

Manages and troubleshoots self-hosted Firecrawl instances for high-performance web-to-markdown scraping.

Acerca de

A comprehensive skill for deploying, maintaining, and automating a self-hosted Firecrawl environment. It provides a robust architecture using Docker Compose, a custom Bun-based scraper wrapper for easier file management, and Caddy for serving scraped results. This skill specifically addresses common reliability issues like container restart policies and connectivity over ZeroTier, ensuring your AI-driven data collection pipeline remains operational and efficient for LLM-ready data extraction.

Características Principales

Pre-configured health checks and troubleshooting routines for API, Redis, and Playwright services.
Integrated Bun wrapper for converting web content directly to markdown files via a simple API.
Automated deployment and configuration of the full Firecrawl stack via Docker Compose.
Optimized restart policies and YAML anchors to ensure service high availability and resilience.
Standardized systemd service management for Caddy file servers and scraper utilities.
9 GitHub stars

Casos de Uso

Converting complex websites into clean markdown for LLM ingestion and RAG knowledge base building.
Setting up a private web scraping infrastructure to bypass rate limits and costs of hosted services.
Troubleshooting container failures and connectivity issues in distributed scraping clusters via ZeroTier.

Acerca de

Características Principales

Pre-configured health checks and troubleshooting routines for API, Redis, and Playwright services.
Integrated Bun wrapper for converting web content directly to markdown files via a simple API.
Automated deployment and configuration of the full Firecrawl stack via Docker Compose.
Optimized restart policies and YAML anchors to ensure service high availability and resilience.
Standardized systemd service management for Caddy file servers and scraper utilities.
9 GitHub stars

Casos de Uso

Converting complex websites into clean markdown for LLM ingestion and RAG knowledge base building.
Setting up a private web scraping infrastructure to bypass rate limits and costs of hosted services.
Troubleshooting container failures and connectivity issues in distributed scraping clusters via ZeroTier.