Does this skill support different storage formats?

Yes, it includes implementation patterns for both SQLite-based caches and Pickle-based file caching to provide flexibility in how data is persisted.

How does this improve performance?

By fetching only new bars (e.g., 13 bars instead of 4 years of data), it reduces API latency from several seconds or minutes to milliseconds for local retrieval.

What is gap-filling in this context?

Gap-filling is the process of identifying the latest timestamp in your local cache and only fetching new data from the API to fill the interval between that timestamp and the current time.

Can this be used for non-trading data?

While optimized for market data, this pattern is applicable to any immutable time-series data where updates only occur at the 'edge' of the dataset.

Why is time-based cache expiry removed?

Historical market data is immutable, meaning past prices do not change. Traditional TTL expiry leads to wasteful re-downloads of data that is already correctly stored locally.

Persistent Market Data Cache & Gap-Filling

Name: Persistent Market Data Cache & Gap-Filling
Author: smith6jt-cop

bysmith6jt-cop

数据科学与机器学习

Optimizes historical market data retrieval by eliminating redundant downloads through persistent caching and incremental gap-filling.

关于

This skill enhances Claude's ability to manage historical financial data efficiently by replacing traditional time-based cache expiry with a persistent storage model. Since historical OHLCV (Open, High, Low, Close, Volume) data is immutable, this skill ensures that once data is validated and stored, it persists indefinitely, only triggering API calls to fetch missing 'gaps' or new bars. This approach drastically reduces API costs, prevents rate-limiting, and accelerates development workflows for quantitative trading and machine learning applications.

主要功能

Advanced cache statistics tracking including hit rates and gap-fill counts
Incremental gap-filling logic to fetch only missing or new market bars
Immutable historical data persistence via SQLite and Pickle cache layers
0 GitHub stars
Automated merging of cached local data with fresh API results
Removal of redundant TTL expiry for historical time-series data

使用场景

Training machine learning models on large historical datasets without daily re-downloads
Reducing API consumption and costs for trading bots by fetching only recent candles
Accelerating backtesting environments with immediate local data access

关于

主要功能

Advanced cache statistics tracking including hit rates and gap-fill counts
Incremental gap-filling logic to fetch only missing or new market bars
Immutable historical data persistence via SQLite and Pickle cache layers
0 GitHub stars
Automated merging of cached local data with fresh API results
Removal of redundant TTL expiry for historical time-series data

使用场景

Training machine learning models on large historical datasets without daily re-downloads
Reducing API consumption and costs for trading bots by fetching only recent candles
Accelerating backtesting environments with immediate local data access