Can it detect if my ML model has collapsed?

Yes, it includes diagnostic checks for zero-variance or constant predictions and provides remediation guidance for common issues in architectures like BiLSTMs.

How does it handle different market hours for crypto and equities?

The skill provides specific annualization factors, such as sqrt(7) for 24/7 crypto and sqrt(5) for equities, to prevent overstating performance on session-filtered data.

Why does this skill use daily aggregation for Sharpe ratios?

Open Deviation Bars have variable durations and high autocorrelation; daily aggregation restores standard IID assumptions for more accurate risk-adjusted return calculations.

What is the Beyond Hit Rate (BHR) framework?

It is a diagnostic suite that uses entropy, CUSUM, and Lempel-Ziv complexity to ensure a signal has genuine predictable structure rather than just a high hit rate.

Open Deviation Bar Evaluation Metrics

Name: Open Deviation Bar Evaluation Metrics
Author: terrylica

byterrylica

•

Data Science & ML

Evaluates trading signal quality and model performance using advanced statistical metrics tailored for non-IID price-based sampling.

This skill provides a comprehensive evaluation framework for quantitative researchers working with Open Deviation Bars (ODB). It addresses the limitations of standard hit rates by implementing the 'Beyond Hit Rate' (BHR) framework, offering metrics like Probabilistic Sharpe Ratio (PSR), Deflated Sharpe Ratio (DSR), and Lempel-Ziv complexity. It helps developers detect model collapse, account for temporal autocorrelation in crypto and equity markets, and validate signal robustness through regime-aware testing and rigorous annualization adjustments.

Key Features

01Beyond Hit Rate (BHR) framework for measuring signal predictability and entropy

02Advanced Sharpe ratio computation with daily aggregation for non-IID bar sequences

03Statistical validation tests including PSR, DSR, and Minimum Track Record Length (MinTRL)

0440 GitHub stars

05Diagnostic tools for detecting model collapse and regime breaks in ML predictions

06Automatic market-specific annualization for 24/7 crypto and 5-day equity sessions

Use Cases

01Validating machine learning model performance on price-sampled financial data

02Generating detailed statistical evaluation reports for quantitative strategy backtests

03Assessing the robustness of trading signals beyond simple win/loss ratios

Key Features

01Beyond Hit Rate (BHR) framework for measuring signal predictability and entropy

02Advanced Sharpe ratio computation with daily aggregation for non-IID bar sequences

03Statistical validation tests including PSR, DSR, and Minimum Track Record Length (MinTRL)

0440 GitHub stars

05Diagnostic tools for detecting model collapse and regime breaks in ML predictions

06Automatic market-specific annualization for 24/7 crypto and 5-day equity sessions

Use Cases

01Validating machine learning model performance on price-sampled financial data

02Generating detailed statistical evaluation reports for quantitative strategy backtests

03Assessing the robustness of trading signals beyond simple win/loss ratios