Document Recommended Practices for High-Throughput Slurm Workloads

[Premas]

What would you like to see added?

Users running Nextflow/nf-core workflows may submit a large number of jobs in a short period of time, which can create a high volume of job submission requests to the Slurm controller. For such cases, reducing the submission rate help prevent large bursts of requests and reduce the risk of temporarily overloading the Slurm controller.

We do not currently have a documented limit on the number of job submissions allowed within a given time period for this type of workflow. However, we recommend users avoid submitting large bursts of jobs that may impact scheduler responsiveness.

Proposed Guidance

Recommend users configure Nextflow to limit the job submission rate:

executor {
submitRateLimit = '1/sec'
}

This allows Nextflow to submit a maximum of one job per second (up to 60 job submissions per minute).

[Premas]

Relevant link: https://seqera.io/blog/5_tips_for_hpc_users/

[Premas]

There is another common scenario that can increase Slurm controller overhead: users submitting multiple jobs through a shell loop. For example:

for i in {1.... n}
do
    sbatch run_mmpbsa.sh $i
done

While the above approach works, it creates many independent jobs and increases scheduler overhead. When jobs are independent and differ only by an index, input file, or output file, Slurm job arrays are the preferred solution.