Is this skill safe for my LLM context window?

Yes. For agentic workflows, you can use semantic instructions and chunks_per_source to return only the most relevant text fragments instead of entire pages.

Can I limit the crawl to specific parts of a website?

Absolutely. You can use regex patterns in the select_paths or exclude_paths fields to target specific directories like /docs or /api.

Do I need an API key to use the Crawl skill?

Yes, this skill requires a Tavily API key. You can get one at tavily.com and add it to your Claude settings file.

What is the difference between Crawl and Map?

Crawl extracts the full content of pages, while Map simply discovers and lists URLs, which is much faster for understanding site structure.

Web Content Crawler

Name: Web Content Crawler
Author: stevessr

bystevessr

0•

웹 스크래핑 및 데이터 수집

Crawls websites to extract content into local markdown files or optimized chunks for AI context using the Tavily API.

The Web Content Crawler skill enables Claude to browse entire websites, documentation portals, or knowledge bases and convert the content into structured markdown files for offline analysis or RAG workflows. By leveraging the Tavily API, it allows for intelligent, instruction-based crawling that can focus on specific technical sections while automatically filtering out noise. It is an essential tool for developers needing to ingest external documentation, perform deep web research, or build local knowledge bases directly within their coding environment without writing custom scraping scripts.

주요 기능

010 GitHub stars

02Semantic, instruction-based crawling to extract only relevant page chunks for AI context

03Deep website crawling with configurable depth (1-5) and page limits

04Automatic conversion of web pages to clean, readable markdown format

05URL discovery mode via Map API for fast site mapping without full content downloads

06Advanced path filtering using regex to include or exclude specific site sections

사용 사례

01Downloading entire technical documentation libraries for offline reference and AI training

02Performing targeted research on specific API endpoints or security guides across a domain

03Automating the collection of structured web content for data analysis or content migration

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add stevessr/dotsfiles crawl

For use in Claude.ai and ChatGPT

Download Skill