Can I use wildcards with the squeue -n flag?

No, the squeue -n flag performs an exact name match. To use pattern matching, pipe the squeue output to grep or a similar text processing utility.

How do I prevent variables from resetting in a bash while loop?

Piping output into a loop (cmd | while) creates a subshell where variable changes are lost. Use process substitution (while read; do ... done < <(cmd)) to run the loop in the main shell.

What is the best way to handle large-scale data transfers on SLURM?

Implement throttling by checking current job counts and use idempotent marker files (.staged) to allow the script to resume safely after interruptions or failures.

Why does my rsync command flatten subdirectories?

Using bash globs like 'source/*/' expands the file list before rsync executes, causing it to copy contents rather than directories. Use the directory path with a trailing slash and specific include/exclude filters to preserve structure.

HPC Batch Staging & Rsync Patterns

Name: HPC Batch Staging & Rsync Patterns
Author: smith6jt-cop

bysmith6jt-cop

•

数据科学与机器学习

Optimizes large-scale data staging on HPC environments using rsync, bash, and SLURM to ensure data integrity and script reliability.

This skill provides specialized guidance for staging massive datasets, such as CODEX raw data, across High-Performance Computing (HPC) storage tiers. It addresses critical failure points in shell scripting, including rsync directory flattening caused by improper glob expansion, variable loss within bash subshell pipes, and incorrect SLURM job monitoring patterns. By utilizing these best practices, users can implement idempotent, throttled data transfers that preserve complex directory structures and maintain accurate job tallies during large-scale batch processing.

主要功能

01Process substitution patterns to prevent variable loss in bash loops

02Pattern-based SLURM job monitoring and throttling logic

03Optimized SLURM resource allocation for high-bandwidth I/O tasks

04Idempotent staging workflows with marker file implementation

05Structure-preserving rsync filter rules for recursive transfers

061 GitHub stars

使用场景

01Transferring multi-terabyte scientific datasets between HPC storage tiers

02Automating throttled batch job submissions for data preprocessing

03Debugging silent data corruption in complex rsync migration scripts

What are Skills?·How to Install

Install with 🐟 Skill.Fish

npx skillfish add smith6jt-cop/skills_registry batch-staging-rsync-patterns

For use in Claude.ai and ChatGPT

Download Skill