How does this skill detect optimization opportunities?

It analyzes your Rust source code for DataFusion context calls, SQL strings, and DataFrame API usage, comparing them against established high-performance patterns and optimization checklists.

Does it support both SQL and the Rust DataFrame API?

Yes, it provides tailored optimization guidance for both the DataFusion SQL interface and the programmatic Rust DataFrame API, including specific syntax for each.

Can it help with Out of Memory (OOM) issues in DataFusion?

Absolutely. The skill identifies common anti-patterns like collecting large results prematurely and suggests configuration changes for memory limits and streaming execution.

How do I verify the improvements suggested by the skill?

The advisor provides instructions on how to use 'EXPLAIN' to inspect logical and physical plans, allowing you to verify that filters are pushed to the storage layer and partitions are being pruned.

DataFusion Query Advisor

Name: DataFusion Query Advisor
Author: EmilLindfors

byEmilLindfors

数据库管理

Optimizes Apache DataFusion SQL queries and DataFrame operations through advanced performance analysis and best practices.

关于

The DataFusion Query Advisor is a specialized skill designed for data engineers and Rust developers working with the Apache DataFusion query engine. It proactively identifies performance bottlenecks in SQL queries and DataFrame code, offering actionable suggestions for optimizations such as predicate pushdown, partition pruning, and column projection. By analyzing query structures and suggesting 'EXPLAIN' plan investigations, this skill helps users minimize I/O overhead, reduce memory consumption, and significantly accelerate data processing tasks in Rust-based data engineering pipelines.

主要功能

Automated identification of inefficient SQL and DataFrame patterns
I/O reduction strategies through smart column projection
0 GitHub stars
Optimization guidance for predicate pushdown and partition pruning
Memory management and parallelism configuration tuning
Detailed code rewrites for performance-critical DataFusion operations

使用场景

Auditing query plans to ensure efficient join ordering and filter placement
Optimizing slow-running analytical queries on large-scale Parquet datasets
Refactoring Rust code to resolve Out of Memory (OOM) errors during data processing

关于

主要功能

Automated identification of inefficient SQL and DataFrame patterns
I/O reduction strategies through smart column projection
0 GitHub stars
Optimization guidance for predicate pushdown and partition pruning
Memory management and parallelism configuration tuning
Detailed code rewrites for performance-critical DataFusion operations

使用场景

Auditing query plans to ensure efficient join ordering and filter placement
Optimizing slow-running analytical queries on large-scale Parquet datasets
Refactoring Rust code to resolve Out of Memory (OOM) errors during data processing