Does this skill support JavaScript-heavy websites?

Yes, it includes specific implementation patterns for JavaScript rendering using PySpider's fetch_type='js' setting and PhantomJS support.

What is the PySpider Development Toolkit for Claude Code?

It is a specialized skill that provides templates and best practices to help developers build reliable, production-ready web crawlers using the PySpider framework.

How does it handle crawler blocking and retries?

The toolkit provides enterprise-grade patterns for proxy rotation, user-agent management, and exponential backoff to ensure crawlers remain operational even on unstable sites.

Why is the on_message method preferred over on_result?

PySpider does not natively support on_result for data storage; this skill enforces the use of on_message to ensure reliable data persistence and inter-project communication.

Can I use this for basic HTML scraping?

Absolutely. It includes a 'Standard Template' specifically designed for simple GET-based HTML scraping tasks with minimal configuration.

PySpider Development Toolkit

Name: PySpider Development Toolkit
Author: Within-7

byWithin-7

Web Scraping & Data Collection

Streamlines the creation of production-ready PySpider web crawlers using standardized templates, advanced scraping patterns, and enterprise-grade best practices.

About

This skill provides a comprehensive framework for building and maintaining robust PySpider crawlers. It offers domain-specific guidance on handling complex scraping scenarios such as JavaScript rendering, proxy rotation, authentication, and multi-page pagination. By enforcing a structured workflow and highlighting critical anti-patterns—such as avoiding blocking operations and correctly utilizing the on_message method—it ensures developers create reliable, scalable data collection pipelines that adhere to enterprise standards.

Key Features

Production-ready patterns for proxy rotation, error handling, and exponential backoff
Standardized data extraction architecture using the BaseHandler and on_message patterns
0 GitHub stars
Technical guidance for JavaScript rendering via fetch_type='js' and PhantomJS
Support for complex pagination, cookie-based authentication, and form handling
Automated crawler initialization with standard, advanced, and enterprise templates

Use Cases

Building high-reliability enterprise web scrapers that require proxy rotation and error recovery
Implementing data collection tasks for dynamic websites involving authentication and AJAX
Migrating basic scraping scripts into a scalable, asynchronous PySpider project structure

About

Key Features

Production-ready patterns for proxy rotation, error handling, and exponential backoff
Standardized data extraction architecture using the BaseHandler and on_message patterns
0 GitHub stars
Technical guidance for JavaScript rendering via fetch_type='js' and PhantomJS
Support for complex pagination, cookie-based authentication, and form handling
Automated crawler initialization with standard, advanced, and enterprise templates

Use Cases

Building high-reliability enterprise web scrapers that require proxy rotation and error recovery
Implementing data collection tasks for dynamic websites involving authentication and AJAX
Migrating basic scraping scripts into a scalable, asynchronous PySpider project structure