Does this skill support parallel processing?

Yes, the --parallel flag allows you to set the number of concurrent threads to significantly speed up large-scale scraping tasks.

Can I restrict scraping to specific domains?

Absolutely. Use the --allowed flag to define which domains the scraper is permitted to visit, ensuring it doesn't wander off to external sites.

How do I find specific information in the scraped files?

Once the scraping is complete, you can use standard CLI tools like grep within the output folder to locate specific patterns or keywords.

What format is the scraped data saved in?

The data is saved as markdown files in a specified output directory, making it easy for Claude to read, search, and parse the content.

Can I limit the depth of a recursive crawl?

Yes, you can use the --max flag to specify the maximum depth of links the scraper should follow from the starting URL.

Web Scraper & Data Extractor

Name: Web Scraper & Data Extractor
Author: AstraBert

byAstraBert

•

网络抓取与数据收集

Extracts web content into structured markdown files using recursive and parallel scraping techniques.

This skill integrates the scpr CLI tool directly into Claude, enabling the agent to crawl websites, capture data, and convert web pages into searchable markdown files. It supports single-page extraction and recursive scraping within specific domains, featuring customizable depth levels and multi-threaded parallel execution to speed up data collection. This is an essential utility for developers and researchers who need Claude to analyze documentation, gather competitive intelligence, or build local knowledge bases from online sources efficiently.

主要功能

01Converts web content directly into clean markdown format

02Integrated workflow for scanning and searching scraped content

03Configurable crawling depth and domain-restricted security filters

04Single-page and recursive domain-wide scraping

059 GitHub stars

06Multi-threaded parallel execution for high-speed data harvesting

使用场景

01Building local knowledge bases from technical guides and blogs

02Gathering research data from multiple web pages for project briefs

03Automated documentation harvesting for offline AI analysis

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add astrabert/scpr web-scraping

For use in Claude.ai and ChatGPT

Download Skill