user testing task-based execution

[atravitz]

We need to cover:

• executing all protocol types
• resuming
• running locally and HPC
• generally try to break things in a practical way (moving the task db and warehouse to a different directory, etc.)

[IAlibay]

[enhancement]

One thing that seems to be missing is how to resume a halted simulation.

[hannahbaumann]

Documentation

• Since the vocabulary of tasks and task-based execution will be new to a lot of our users, it would be great to introduce them in the user guide.

• This part in the user guide is confusing, since the default alchemical network output file name is alchemicalNetwork.json, but here we're switching to tyk2.json, even though the paragraph above says the output is alchemicalNetwork.json.

  Once you have an AlchemicalNetwork, use the following command to set up the task-based campaign. By default, the TaskDB and Warehouse will be created using the input file basename (here, tyk2).

• Re parallel submission of multiple jobs: I think here we could also mention the use of array jobs, e.g. sbatch --array=1-100 run_tasks.sh which would make parallel execution more practical then manually submitting the same job many times.

• Maybe the user guide could also briefly explain where run-task stores the results? (including the shared and scratch directories?)

Questions

• A use case I was thinking about is networks with a mix of charge-changing and neutral edges. If I know that the charge-changing transformations take ~3x longer, could I somehow identify/filter those tasks and submit them in separate slurm scripts, e.g. to a better GPU or with a longer wall time using some metadata or tags, e.g. --task-class expensive?

• Is it be possible to use openfe run-task together with MPS, potentially only for certain tasks, e.g. solvent legs (see above Re tags)?

• In the first example I ran, all my setup units ran fine, but the first simulation run unit failed because of an incompatible CUDA version (marked as TOO_MANY_RETRIES). After downgrading the cuda version the second simulation unit runs successfully.
  Is there a way to reset/requeue just that failed task so a worker picks it up again? Or would deleting the failed output be enough for it to get resubmitted automatically?

Different things

• Re --networks-only: Would this be a good time to change the default folder name alchemicalNetwork? It's confusing since it also contains the ligand network, and also the capital N is not great. Or would that be API breaking?
• Nice to have for the future: If a specific unit failed (seen in the the openfe status --task-db), it's currently not easy for the user to know which edge that unit belongs to. This is a bit different from quickrun, where the ligand names would usually be in the folder name containing all units for that transformation. Maybe some helper to easily map a unit back to its edge would be useful?

[jthorton]

Setup:

• Can we have an option to name the warehouse and task db?
• Is it possible to add more tasks to a db once created? Thinking about the broken network project how would a user add them to keep them organised in one place?

Error handling:

• Setting up a task campaign for a network which has validation issues (raised when create is called) is difficult to debug as its not clear which transformation caused the error. Tested with the attached network which has two broken transformations only the first is raised as an error.

Question:

• How does the user work out which transformations have successfully completed or link an errored unit to the transformation which has failed?

alchemical_network.json