Optimizes and builds high-performance distributed data processing pipelines using Apache Spark and PySpark.
The Spark Engineer skill transforms Claude into a senior big data specialist focused on the Apache Spark ecosystem. It provides expert guidance on building scalable ETL pipelines, optimizing Spark SQL and DataFrame operations, and managing complex streaming analytics. Whether you are troubleshooting memory issues, tuning shuffle partitions, or implementing data salting to handle skew, this skill ensures your distributed computing workloads are production-grade, efficient, and cost-effective.
주요 기능
01Optimized DataFrame and Spark SQL implementation
0249 GitHub stars
03Intelligent data partitioning and caching strategies
04Resource management and memory troubleshooting
05Advanced performance tuning and shuffle optimization
06Production-grade Structured Streaming patterns
사용 사례
01Developing petabyte-scale ETL pipelines with PySpark or Scala
02Migrating legacy RDD operations to high-performance DataFrame APIs
03Optimizing slow Spark jobs by identifying and resolving data skew
