Pipeline Architecture & ETL Design

关于

This skill enables Claude to guide developers through the implementation of the Pipes and Filters architectural pattern, specifically optimized for high-throughput data processing and transformation tasks. It provides structured guidance on defining stateless filters, establishing reliable pipe connections with back-pressure, and instrumenting each stage for observability. By utilizing this skill, developers can create modular, scalable systems for ETL jobs, streaming analytics, or compiler-like workloads while ensuring failure isolation and horizontal scalability across individual processing stages.

主要功能

• Automated generation of Architecture Decision Records (ADRs)
• Best practices for stateless filter design and data schema definition
• Observability and monitoring strategies for multi-stage pipelines
• 10 GitHub stars
• Implementation patterns for back-pressure and buffering mechanisms
• Standardized pipes-and-filters architectural guidance

使用场景

• Designing streaming analytics platforms with discrete processing stages
• Developing modular CI/CD pipelines or compiler-like transformation engines
• Building high-performance ETL (Extract, Transform, Load) data jobs