How to increase parallelism of minimap2_wnode_fasta in EGAPx?

[zhoudreames]

Hello EGAPx developers,
I am running EGAPx v0.5.2 on a Slurm cluster. The pipeline works normally, but I noticed that the minimap2_wnode_fasta step has very limited parallelism.
Example:

ega…minimap2_wnode_fasta (139) | 130 of 2304

Although there are 2304 tasks, only ~1–2 Slurm jobs are running at the same time. Other EGAPx steps (e.g., bam_bin) can run many jobs concurrently, so the Slurm and Nextflow configurations seem to work properly.
My configuration:

minimap2_wnode:
  gpx_qsubmit: -batch-size 5000 -max-execs 100
  split: 16

Could you please advise:

Which parameter controls the number of concurrent jobs for this step?
Should I increase -max-execs or split for large-scale annotation?
Thank you for your help.

Best regards,
Dreames

[etvedte]

Can you provide some more info?

On #261, you mentioned you have a large volume of RNA-seq, which we recommended you sample down quite a bit to get a handle on your HPC configuration. How many short read/long read runs are you using? When you say "The pipeline works normally," have you completed any EGAPx runs with your downsampled data?

What other tasks are running concurrently with minimap2_wnode_fasta? The default minimap2 parameters are being set here which means more than two processes can be running at once, but it's possible there are no available hosts depending on what else is going on concurrently. You gave the example of bam_bin running many jobs concurrently, and since that is a task that runs on a single cpu the slurm scheduler might be prioritizing those jobs.

If you are willing to share some of your configuration files (e.g. your process_resources.config and slurm.config files) and share some of the details of the processes that are going on concurrently that might help shed light on the problem.

[zhoudreames]

Can you provide some more info?

On #261, you mentioned you have a large volume of RNA-seq, which we recommended you sample down quite a bit to get a handle on your HPC configuration. How many short read/long read runs are you using? When you say "The pipeline works normally," have you completed any EGAPx runs with your downsampled data?

What other tasks are running concurrently with minimap2_wnode_fasta? The default minimap2 parameters are being set here which means more than two processes can be running at once, but it's possible there are no available hosts depending on what else is going on concurrently. You gave the example of bam_bin running many jobs concurrently, and since that is a task that runs on a single cpu the slurm scheduler might be prioritizing those jobs.

If you are willing to share some of your configuration files (e.g. your process_resources.config and slurm.config files) and share some of the details of the processes that are going on concurrently that might help shed light on the problem.

Thank you for your response. Here are some additional details:

• I am using 140 long-read runs, each downsampled to 3 million reads.
• I am using 200 short-read RNA-seq runs, each downsampled to 10 million read pairs.
• I have successfully completed an EGAPx test run using a single chromosome with the downsampled data.

My main goal is to increase the parallelism of minimap2_wnode_fasta, since my HPC cluster has sufficient available resources. Currently, this step uses less than 30% of the cluster’s available capacity.

Interestingly, other EGAPx processes, such as bin_and_sort:merge and bam2asn:convert, can gradually submit up to approximately 100 concurrent jobs. In contrast, minimap2_wnode_fasta remains limited to around 16 concurrently running jobs. This also occurs after all other tasks have finished, when minimap2_wnode_fasta is the only active process. Therefore, resource competition from other EGAPx tasks does not appear to explain the limitation.

For example:

[2f/4869af] ega…_bin_and_sort:merge (1260) | 1260 of 1260, cached: 1260 ✔
[08/1b852a] ega…ane:bam2asn:convert (1106) | 1260 of 1260, cached: 1259 ✔
[de/83cd8b] ega…seq_collapse:generate_jobs | 1 of 1 ✔
[57/ea0b4e] ega…se:run_rnaseq_collapse (1) | 10 of 11, failed: 1, retries: 1
[2e/db10f9] ega…2_index:run_minimap2_index | 1 of 1, cached: 1 ✔
[7b/0e94e8] ega…ane:rename_fasta_ids (142) | 144 of 144, cached: 144 ✔
[9c/c600fd] ega…ta:split_fasta_files (142) | 144 of 144, cached: 144 ✔
[da/9b3604] ega…minimap2_wnode_fasta (345) | 335 of 2304

My cluster has enough resources to run approximately 100 minimap2_wnode_fasta jobs concurrently, so I would like to understand what is limiting this process to around 16 jobs and which configuration parameter should be adjusted to increase its parallelism.

I have attached my process_resources.config and slurm.config files for reference. Please let me know if any additional logs or configuration files would be helpful.

slurm.cg.txt

process_resources.cg.txt

[etvedte]

@zhoudreames - edited response below

minimap2_wnode_fasta currently has a hard cap of 16 concurrent processes defined by maxForks here. You can test increasing that with your resources configuration to increase cluster allocation.

Based on #261 (particularly the node count) you could have:

params.threads = 32
params.nodes = 10
params.num_cpus_per_node = 96

How the jobs are split within/across nodes are  'hopes', not 'promises'. EGAPx has a deterministic way of splitting up jobs, but what nextflow will do with those jobs is seemingly not deterministic.