Proposal: Firedrake-ify the pyadjoint repo

[connorjward]

In a recent Firedrake meeting we discussed the possibility of allowing firedrake-zenodo (https://www.firedrakeproject.org/zenodo.html) to work with pyadjoint. This means allowing the script to make prereleases such that Zenodo may give them DOIs for users to cite.

Along similar lines it would also be convenient for us to have pyadjoint follow the same release process as the rest of Firedrake (excluding UFL). This would involve moving to release and main branches instead of a single master branch.

@jorgensd do you have any objections to either of these?

@dham FYI

[jorgensd]

This would involve moving to release and main branches instead of a single master branch.

I am not super fond of have to branches that needs to be maintained in parallel. However, as you guys are the once that are most active in this repository, I wont stand in the way of that.

Along similar lines it would also be convenient for us to have pyadjoint follow the same release process as the rest of Firedrake (excluding UFL).

Very happy to have formal release guidelines for the repo

In a recent Firedrake meeting we discussed the possibility of allowing firedrake-zenodo (https://www.firedrakeproject.org/zenodo.html) to work with pyadjoint. This means allowing the script to make prereleases such that Zenodo may give them DOIs for users to cite.

However, I don't know what you would need to integrate into pyadjoint to make this work? (I see that the script is available at: https://github.com/firedrakeproject/firedrake/blob/main/firedrake/scripts/firedrake-zenodo but haven't had time to inspect it in detail).

Could you elaborate on what this would add to pyadjoint, and how this would interact with firedrake (as we don't want to have a firedrake dependency inside pyadjoint). If you think the proposed workflow for tracking dependencies in firedrake-zenodo is a good general purpose tool, it should probably be separated from the firedrake repo and serve as a general tool that can be configured to work with firedrake (or FEniCS or whoever uses pyadjoint) with their computational stack.

[connorjward]

I am not super fond of have to branches that needs to be maintained in parallel. However, as you guys are the once that are most active in this repository, I wont stand in the way of that.

I find it helpful to have a clear way to distinguish between a bug fix branch and a main development branch. If one needs to make a patch release then the process is clear whereas if there isn't an existing branch then it's more complicated. Do you prefer a different system?
This also is not a critical point. It is just convenient to have similar workflows everywhere.

However, I don't know what you would need to integrate into pyadjoint to make this work? (I see that the script is available at: https://github.com/firedrakeproject/firedrake/blob/main/firedrake/scripts/firedrake-zenodo but haven't had time to inspect it in detail).

Nothing needs changing inside pyadjoint. The firedrake-zenodo script would just create additional GitHub releases corresponding to the commits that users have used in their research. If they use an existing release of pyadjoint then no new release is made. I'm basically just asking permission to muddy the GitHub releases section of this repository with prereleases for users to cite.

If you think the proposed workflow for tracking dependencies in firedrake-zenodo is a good general purpose tool, it should probably be separated from the firedrake repo and serve as a general tool that can be configured to work with firedrake (or FEniCS or whoever uses pyadjoint) with their computational stack.

I have wanted to do this for some time. However I haven't quite thought through exactly how one might do that and I certainly don't have enough time for that in the near future.

[jorgensd]

Fair enough. Some follow-up questions:

Could you explain to me who would have access to run the script that would add the releases? Would every user be able to push a tag to the pyadjoint repo, or only a select few maintainers?

Would this in theory mean that every commit would have an associated release? If yes, why not rebase the workflow based on latest release + commit hash?

Would it only work for commits/branches from the pyadjoint main repo?

[connorjward]

Could you explain to me who would have access to run the script that would add the releases? Would every user be able to push a tag to the pyadjoint repo, or only a select few maintainers?

You need to use a GitHub API token. So only people with the permission to make pyadjoint releases are able to do it.

Would this in theory mean that every commit would have an associated release? If yes, why not rebase the workflow based on latest release + commit hash?

I don't quite understand what you mean by this. Every commit can have an associated release, but only if a user says that they used that commit in their research and I/another maintainer ran the firedrake-zenodo script to create it.

Would it only work for commits/branches from the pyadjoint main repo?

Yep. Only commits on the pyadjoint repo can be used.

[jorgensd]

Could you explain to me who would have access to run the script that would add the releases? Would every user be able to push a tag to the pyadjoint repo, or only a select few maintainers?

You need to use a GitHub API token. So only people with the permission to make pyadjoint releases are able to do it.

So it would mainly be useful for the pyadjoint maintainers? If so, why not just more frequent releases?

Would this in theory mean that every commit would have an associated release? If yes, why not rebase the workflow based on latest release + commit hash?

I don't quite understand what you mean by this. Every commit can have an associated release, but only if a user says that they used that commit in their research and I/another maintainer ran the firedrake-zenodo script to create it.

What I am worried about is having alot of in-between release tags that muddles the history (and changelog?) between full releases.
As you or another maintainer have to run it, I don't see the need for an intermediate tag.
Maybe I'm just a bit slow at grasping the benefit of this approach.

[connorjward]

I'm afraid I haven't explained things very clearly. The Zenodo process is fairly weird:

1. A user wants to provide references for the software that they used to run their simulation. They therefore run firedrake-zenodo themselves. This generates a JSON file with information about which versions of the software they used. This means that for Firedrake, FIAT, PETSc, etc the JSON will contain either a version string or a commit hash.
2. The user then gets in touch with us via an issue where they provide this JSON file and ask for a Zenodo release (example).
3. We (the maintainers) then run firedrake-zenodo a second time. This time it reads the JSON file and makes sure that there are GitHub releases that correspond to those version numbers/commit hashes. If releases don't exist then we create them.
4. These releases get detected by Zenodo and DOIs are created for them.
5. We then create a 'meta' release that gathers them together (not relevant to this discussion).

What I am worried about is having alot of in-between release tags that muddles the history (and changelog?) between full releases.

As far as possible we try to avoid doing this. The Zenodo-specific releases are only prereleases and I can guarantee that the changelog won't be affected.

[jorgensd]

Ok, then we can try this out and see how it goes:)