Rewrite auth challenge on repository paths, not just /v2/ - #91
Open
rebkwok wants to merge 1 commit into
Open
Conversation
The WWW-Authenticate rewrite that points clients at this proxy instead
of ghcr.io was only applied in `location = /v2/`, and assumed that any
auth challenge comes to /v2/.
Only clients that establish auth from the /v2/ have ever been
served correctly. Clients that read the challenge from repository
endpoints, which the registry auth flow permits and which containerd's
resolver has always done, are sent to ghcr.io instead. That gap was
invisible while every backend used Docker's legacy pull path, which
pings /v2/ first and caches the challenge per registry host.
Docker Engine 29 makes the containerd image store the
default on fresh installs, so newly provisioned hosts hit the
unrewritten challenge and fail, while hosts upgraded into 29 keep the
legacy graph driver and keep working.
This was identified with a test emisv2 backend instance, using the
proxy on dokku-emis-staging; it is running Docker v29.1.3 but a
fresh install and hits this issue.
The TPP backend is ALSO running Docker v29.1.3, but was upgraded from
an earlier version and so is continuing to use the legacy driver.
docker info --format 'Server {{.ServerVersion}} / Driver {{.Driver}} {{.DriverStatus}}'
On the new test backend instance:
Server 29.1.3 / Driver overlayfs [[driver-type io.containerd.snapshotter.v1]]
On TPP backend:
Server 29.1.3 / Driver overlay2 [[Backing Filesystem extfs] [Supports
d_type true] [Using metacopy false] [Native Overlay Diff false]
[userxattr false]]
TPP is on v29; if we every rebuild it, it will also break.
Apply the same proxy_hide_header/add_header pair to the
/v2/opensafely-core/ location, so we now cover both possibilities.
.
Note the /token passthrough needs no change: ghcr.io
accepts the rewritten service parameter and returns a usable token.
have error_page handlers.
Refs:
https://docs.docker.com/engine/storage/containerd/
https://www.docker.com/blog/docker-engine-version-29/
Contributor
Author
|
I've build from this branch and deployed the dokku-emis-staging proxy with it, and confirm that it is now working (and the test backend instance has picked up the jobs that I tried to run on it last week and has now completed them). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The WWW-Authenticate rewrite that points clients at this proxy instead of ghcr.io was only applied in
location = /v2/, and assumed that any auth challenge comes to /v2/.Only clients that establish auth from the /v2/ have ever been served correctly. Clients that read the challenge from repository endpoints, which the registry auth flow permits and which containerd's resolver has always done, are sent to ghcr.io instead. That gap was invisible while every backend used Docker's legacy pull path, which pings /v2/ first and caches the challenge per registry host.
Docker Engine 29 makes the containerd image store the default on fresh installs, so newly provisioned hosts hit the unrewritten challenge and fail, while hosts upgraded into 29 keep the legacy graph driver and keep working.
This was identified with a test emisv2 backend instance, using the proxy on dokku-emis-staging; it is running Docker v29.1.3 but a fresh install and hits this issue.
The TPP backend is ALSO running Docker v29.1.3, but was upgraded from an earlier version and so is continuing to use the legacy driver.:
On the new test backend instance:
Server 29.1.3 / Driver overlayfs [[driver-type io.containerd.snapshotter.v1]]On TPP backend:
TPP is on v29; if we every rebuild it, it will also break.
Apply the same proxy_hide_header/add_header pair to the /v2/opensafely-core/ location, so we now cover both possibilities. .
Note the /token passthrough needs no change: ghcr.io accepts the rewritten service parameter and returns a usable token. have error_page handlers.
Refs:
https://docs.docker.com/engine/storage/containerd/
https://www.docker.com/blog/docker-engine-version-29/