Does this skill support JavaScript-heavy websites?

Yes, it includes specific implementation patterns for JavaScript rendering using PySpider's fetch_type='js' setting and PhantomJS support.

What is the PySpider Development Toolkit for Claude Code?

It is a specialized skill that provides templates and best practices to help developers build reliable, production-ready web crawlers using the PySpider framework.

How does it handle crawler blocking and retries?

The toolkit provides enterprise-grade patterns for proxy rotation, user-agent management, and exponential backoff to ensure crawlers remain operational even on unstable sites.

Why is the on_message method preferred over on_result?

PySpider does not natively support on_result for data storage; this skill enforces the use of on_message to ensure reliable data persistence and inter-project communication.

Can I use this for basic HTML scraping?

Absolutely. It includes a 'Standard Template' specifically designed for simple GET-based HTML scraping tasks with minimal configuration.

PySpider Development Toolkit

Name: PySpider Development Toolkit
Author: Within-7

byWithin-7

0•

웹 스크래핑 및 데이터 수집

Streamlines the creation of production-ready PySpider web crawlers using standardized templates, advanced scraping patterns, and enterprise-grade best practices.

This skill provides a comprehensive framework for building and maintaining robust PySpider crawlers. It offers domain-specific guidance on handling complex scraping scenarios such as JavaScript rendering, proxy rotation, authentication, and multi-page pagination. By enforcing a structured workflow and highlighting critical anti-patterns—such as avoiding blocking operations and correctly utilizing the on_message method—it ensures developers create reliable, scalable data collection pipelines that adhere to enterprise standards.

주요 기능

01Production-ready patterns for proxy rotation, error handling, and exponential backoff

02Standardized data extraction architecture using the BaseHandler and on_message patterns

030 GitHub stars

04Technical guidance for JavaScript rendering via fetch_type='js' and PhantomJS

05Support for complex pagination, cookie-based authentication, and form handling

06Automated crawler initialization with standard, advanced, and enterprise templates

사용 사례

01Building high-reliability enterprise web scrapers that require proxy rotation and error recovery

02Implementing data collection tasks for dynamic websites involving authentication and AJAX

03Migrating basic scraping scripts into a scalable, asynchronous PySpider project structure

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add within-7/minto-plugin-tools skills

For use in Claude.ai and ChatGPT

Download Skill