Is this skill compatible with Snakemake 8.0?

Yes, it is designed for Snakemake 8.0 and above, specifically leveraging the snakemake-executor-plugin-slurm for modern cluster interactions.

Can this architecture handle both GPU and CPU jobs?

Yes, it supports both. However, it prioritizes GPU-only scheduling for intensive tasks like BaSiC fitting to avoid the 15-20x performance penalty of CPU fallbacks.

How does this skill handle multi-account SLURM routing?

It uses lambda functions within the Snakemake 'resources' block to dynamically assign the correct slurm_account and partition based on the job's requirements and wildcard values.

Why are sentinel files used instead of standard file outputs?

In workflows like image stitching that produce hundreds of files per cycle, declaring every file as an output creates a fragile and slow DAG. Sentinel files provide a single source of truth for step completion.

What happens if a multi-channel job is interrupted?

The workflow includes internal wrapper logic that checks for completed channels before processing, allowing jobs to resume exactly where they left off even if the Snakemake sentinel wasn't written.

Snakemake SLURM Workflow Architecture

Name: Snakemake SLURM Workflow Architecture
Author: smith6jt-cop

bysmith6jt-cop

•

デプロイメントとDevOps

Orchestrates high-performance computing workflows on SLURM clusters using Snakemake with intelligent multi-account resource routing.

This skill provides a standardized architectural pattern for migrating legacy HPC shell scripts to declarative Snakemake workflows. It specializes in solving the 'multi-account routing' problem where jobs must be dispatched to specific SLURM accounts based on hardware requirements (GPU vs. CPU). It enables Claude to implement advanced features like lambda-driven resource allocation, sentinel-based output tracking for massive datasets, and aggregate QC reporting. This is ideal for complex bio-imaging or data science pipelines requiring deterministic scheduling and robust error recovery on clusters like HiPerGator.

主な機能

01Aggregate QC and registration rules for cross-cycle data alignment

02GPU-optimized cycle pre-assignment for 15-20x performance gains

031 GitHub stars

04Dynamic multi-account SLURM routing via lambda resource functions

05Internal per-channel skip-existing logic for efficient job resumption

06Sentinel file management for tracking hundreds of per-cycle outputs

ユースケース

01Implementing robust automated QC reporting for multi-stage scientific workflows

02Migrating complex monolithic bash submission scripts to declarative Snakemake pipelines

03Managing resource-intensive image processing pipelines across multiple HPC GPU accounts

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add smith6jt-cop/skills_registry snakemake-workflow-architecture

For use in Claude.ai and ChatGPT

Download Skill