How does the caching system work?

It uses a local SQLite database to store scraped pages for 7 days. This prevents redundant network requests, avoids hitting rate limits, and speeds up subsequent runs.

Can I control the depth of the web crawl?

Yes, the --spider-depth option allows you to control the crawl: depth 0 scrapes only the seed URL, depth 1 includes direct links, and depth 2 goes one level deeper.

What documentation can I scrape with this skill?

The skill is specifically optimized for docs.snowflake.com. You can target any section, such as SQL references or migration guides, using the --base-path parameter.

Where are the scraped files stored?

Files are saved to your specified output directory, organized by the URL structure, and include a generated SKILL.md index for easy navigation.

Snowflake Documentation Scraper

Name: Snowflake Documentation Scraper
Author: sfc-gh-dflippo

bysfc-gh-dflippo

•

Web Scraping & Data Collection

Extracts and organizes Snowflake documentation into structured Markdown format with intelligent caching and configurable spider depth.

The doc-scraper skill is a specialized utility designed to ingest and organize technical content from the Snowflake documentation site. It automates the process of converting web pages into clean, searchable Markdown files complete with metadata frontmatter, making it an essential tool for developers building local knowledge bases or providing high-quality technical context to AI models. With built-in SQLite caching that expires every seven days and a configurable spider engine, it ensures efficient data collection while respecting rate limits and minimizing redundant network traffic.

Key Features

01Intelligent SQLite caching with 7-day expiration cycles

02Configurable rate limiting and concurrent thread management

03Automated Markdown conversion with structured frontmatter

04Configurable spider depth for targeted or deep-crawl scraping

0530 GitHub stars

06Base-path filtering to target specific documentation sections

Use Cases

01Offline access to specific Snowflake migration guides and architectural patterns

02Preparing technical documentation for Retrieval-Augmented Generation (RAG) workflows

03Generating a local, searchable Markdown library of Snowflake SQL references

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add sfc-gh-dflippo/snowflake-dbt-demo doc-scraper

For use in Claude.ai and ChatGPT

Download Skill