Documentation Scraper

概要

The Documentation Scraper skill provides a robust framework for converting web-based documentation into high-quality, searchable reference data. It automates the extraction process using configurable CSS selectors and URL patterns, mapping site content into logical categories like API references, guides, and tutorials. Designed specifically for agentic coding workflows, it includes built-in mitigations for common LLM failure modes, such as grounding checks via robots.txt verification and uncertainty escalation when selectors are ambiguous, ensuring reliable output for building Claude skills or local knowledge bases.

主な機能

• Configurable CSS selectors for precise extraction of main content, titles, and code blocks
• Automatic content categorization using keyword mapping and directory structures
• Pattern-based URL filtering to include or exclude specific documentation paths
• 67 GitHub stars
• Resumable progress tracking via a dedicated checkpoint system for large-scale scrapes
• Agentic safety protocols including rate limiting, robots.txt compliance, and grounding checks

ユースケース

• Creating offline, structured references for complex API documentation to reduce token usage
• Standardizing documentation from multiple web sources into a unified markdown format for AI training
• Building custom Claude Code skills by scraping official library or framework documentation