What is the 'Zero-Field-Loss' principle?

It is a strict refactoring redline ensuring that no data fields or API parameters are removed during optimization, prioritizing data integrity over code elegance.

How does the skill manage database operations?

It includes automation scripts for registering crawler mappings in MongoDB, configuring field projections for Excel exports, and performing clean project deletions.

Why are strict exceptions required instead of logger errors?

To ensure that failures trigger a 'FAILED' status within the Pyspider lifecycle, which is essential for automated monitoring, retries, and webhook notifications.

What are the Strategy Patterns A-E included in this skill?

They are specialized templates for different scraping needs: Strategy A (BrightData V3), B (Cookie Pools), C (SSR/Regex), D (API Forwarding), and E (Dispatcher for task routing).

Production Pyspider Crawler SOP

Name: Production Pyspider Crawler SOP
Author: Within-7

byWithin-7

•

웹 스크래핑 및 데이터 수집

Streamlines Pyspider crawler development and refactoring using production-grade strategy patterns and strict engineering redlines.

This skill provides a comprehensive framework for building and maintaining robust Pyspider web crawlers through standardized SOPs. It offers dual-mode workflows for creating new projects or refactoring existing ones, supported by five distinct strategy patterns ranging from BrightData V3 integration and Cookie pool management to SSR parsing and API forwarding. By enforcing strict engineering redlines—such as the Zero-Field-Loss principle and mandatory exception triggers—it ensures high-quality data collection, reliable task scheduling via MongoDB, and seamless synchronization between scripts, Git repositories, and database configurations.

주요 기능

01Strict engineering redlines to ensure zero data loss and robust error handling.

02Dual-mode workflows for both fresh project generation and production refactoring.

03Automated MongoDB registration and cleanup scripts for crawler management.

04Five specialized strategy templates (A-E) for diverse anti-scraping scenarios.

05Transparent auditing guidelines for optimizing and simplifying legacy crawler code.

061 GitHub stars

사용 사례

01Refactoring legacy Pyspider scripts while maintaining pixel-perfect data field alignment.

02Managing large-scale distributed scraping tasks using a dispatcher-routing architecture.

03Developing high-performance crawlers for sites with advanced anti-scraping measures like BrightData.

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add within-7/minto-plugin-tools production-sop

For use in Claude.ai and ChatGPT

주요 기능

01Strict engineering redlines to ensure zero data loss and robust error handling.

02Dual-mode workflows for both fresh project generation and production refactoring.

03Automated MongoDB registration and cleanup scripts for crawler management.

04Five specialized strategy templates (A-E) for diverse anti-scraping scenarios.

05Transparent auditing guidelines for optimizing and simplifying legacy crawler code.

061 GitHub stars

사용 사례

01Refactoring legacy Pyspider scripts while maintaining pixel-perfect data field alignment.

02Managing large-scale distributed scraping tasks using a dispatcher-routing architecture.

03Developing high-performance crawlers for sites with advanced anti-scraping measures like BrightData.