Skip to content

Rewrite auth challenge on repository paths, not just /v2/ - #91

Open
rebkwok wants to merge 1 commit into
mainfrom
update-ghcr
Open

Rewrite auth challenge on repository paths, not just /v2/#91
rebkwok wants to merge 1 commit into
mainfrom
update-ghcr

Conversation

@rebkwok

@rebkwok rebkwok commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

The WWW-Authenticate rewrite that points clients at this proxy instead of ghcr.io was only applied in location = /v2/, and assumed that any auth challenge comes to /v2/.

Only clients that establish auth from the /v2/ have ever been served correctly. Clients that read the challenge from repository endpoints, which the registry auth flow permits and which containerd's resolver has always done, are sent to ghcr.io instead. That gap was invisible while every backend used Docker's legacy pull path, which pings /v2/ first and caches the challenge per registry host.

Docker Engine 29 makes the containerd image store the default on fresh installs, so newly provisioned hosts hit the unrewritten challenge and fail, while hosts upgraded into 29 keep the legacy graph driver and keep working.

This was identified with a test emisv2 backend instance, using the proxy on dokku-emis-staging; it is running Docker v29.1.3 but a fresh install and hits this issue.
The TPP backend is ALSO running Docker v29.1.3, but was upgraded from an earlier version and so is continuing to use the legacy driver.:

docker info --format 'Server {{.ServerVersion}} / Driver {{.Driver}} {{.DriverStatus}}'

On the new test backend instance:
Server 29.1.3 / Driver overlayfs [[driver-type io.containerd.snapshotter.v1]]

On TPP backend:

Server 29.1.3 / Driver overlay2 [[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff false] [userxattr false]]

TPP is on v29; if we every rebuild it, it will also break.

Apply the same proxy_hide_header/add_header pair to the /v2/opensafely-core/ location, so we now cover both possibilities. .
Note the /token passthrough needs no change: ghcr.io accepts the rewritten service parameter and returns a usable token. have error_page handlers.

Refs:
https://docs.docker.com/engine/storage/containerd/
https://www.docker.com/blog/docker-engine-version-29/

The WWW-Authenticate rewrite that points clients at this proxy instead
of ghcr.io was only applied in `location = /v2/`, and assumed that any
auth challenge comes to /v2/.

Only clients that establish auth from the /v2/ have ever been
served correctly. Clients that read the challenge from repository
endpoints, which the registry auth flow permits and which containerd's
resolver has always done, are sent to ghcr.io instead. That gap was
invisible while every backend used Docker's legacy pull path, which
pings /v2/ first and caches the challenge per registry host.

Docker Engine 29 makes the containerd image store the
default on fresh installs, so newly provisioned hosts hit the
unrewritten challenge and fail, while hosts upgraded into 29 keep the
legacy graph driver and keep working.

This was identified with a test emisv2 backend instance, using the
proxy on dokku-emis-staging; it is running Docker v29.1.3 but a
fresh install and hits this issue.
The TPP backend is ALSO running Docker v29.1.3, but was upgraded from
an earlier version and so is continuing to use the legacy driver.

docker info --format 'Server {{.ServerVersion}} / Driver {{.Driver}} {{.DriverStatus}}'
On the new test backend instance:
  Server 29.1.3 / Driver overlayfs [[driver-type io.containerd.snapshotter.v1]]
On TPP backend:
  Server 29.1.3 / Driver overlay2 [[Backing Filesystem extfs] [Supports
d_type true] [Using metacopy false] [Native Overlay Diff false]
[userxattr false]]

TPP is on v29; if we every rebuild it, it will also break.

Apply the same proxy_hide_header/add_header pair to the
/v2/opensafely-core/ location, so we now cover both possibilities.
.
Note the /token passthrough needs no change: ghcr.io
accepts the rewritten service parameter and returns a usable token.
have error_page handlers.

Refs:
  https://docs.docker.com/engine/storage/containerd/
  https://www.docker.com/blog/docker-engine-version-29/
@rebkwok

rebkwok commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

I've build from this branch and deployed the dokku-emis-staging proxy with it, and confirm that it is now working (and the test backend instance has picked up the jobs that I tried to run on it last week and has now completed them).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant