How can I speed up runs safely?¶
The main controls are:
--n-jobs: number of parallel jobs (pipeline-level parallelism)--threads: threads per job (tool-level parallelism; available in some modules)
Tuning guide¶
- Start with
--n-jobsequal to available CPU cores divided by 2. - Increase gradually while monitoring RAM and disk I/O.
- If the machine becomes unstable (heavy swapping, stalled I/O), reduce
--n-jobsfirst. - Use higher
--threadsonly when the underlying step benefits from multithreading.
Rules of thumb¶
- Many small files/steps: increase
--n-jobs. - Few compute-heavy steps: balance
--n-jobsand--threads. - Shared servers/HPC login nodes: be conservative and follow local policy.
Example¶
1 | |
See also: