Vaex Big Data Processing

关于

The Vaex skill empowers Claude to handle massive tabular datasets that exceed available RAM by leveraging memory mapping and lazy evaluation. It provides expert guidance on processing over a billion rows per second, making it ideal for interactive data exploration, complex statistical aggregations, and building machine learning pipelines on big data. This skill ensures Claude applies best practices for virtual columns, efficient I/O operations with HDF5 and Parquet, and high-speed visualization without the memory overhead common in traditional data libraries.

主要功能

• Out-of-core DataFrame operations for datasets exceeding available RAM
• Integrated machine learning pipelines for out-of-memory datasets
• Memory-efficient virtual columns and lazy evaluation patterns
• Billion-row per second processing speed via memory mapping
• High-speed statistical aggregations and massive-scale visualizations
• 8 GitHub stars

使用场景

• Performing fast feature engineering and ML training on massive datasets
• Processing and cleaning multi-terabyte CSV, HDF5, or Parquet files
• Generating interactive heatmaps and histograms from billions of data points