[Bug]: vminitd: parallel clients of the forwarded ssh-agent hang on live connections; the container's terminal can freeze with them

[maxim-uvarov]

I have done the following

• I have searched the existing issues
• If possible, I've reproduced the issue using the 'main' branch of this project

Steps to reproduce

the report below was generated by using claude

Summary

When several guest processes talk to /var/host-services/ssh-auth.sock at the same time, some of them never get an answer.
The connections are live: the host agent holds them open, no bytes are queued on any host socket, and killing the host agent releases every hung client at once with "communication with agent failed".
Only the data event is lost, on the guest side of the relay.
In two catches on a real machine the container's interactive terminal and every container exec froze a few seconds after the clients started to hang, and came back the moment the host agent was killed.

Steps to reproduce

container run --detach --name t --ssh debian:12 sleep infinity     # any image with ssh-keygen
container exec t sh -c 'apt-get update && apt-get install -y openssh-client'
container exec t sh -c 'ssh-keygen -q -t ed25519 -N "" -f /tmp/k && for i in $(seq 16); do echo hi > /tmp/m.$i; done'

Then rounds of sixteen parallel signatures with the file key.
ssh-keygen -Y sign first asks the agent whether it holds the key, then reads the file, so each sign is one short agent conversation:

for r in $(seq 30); do
  echo "round $r"
  timeout 10 container exec t sh -c 'for i in $(seq 16); do ssh-keygen -Y sign -q -f /tmp/k -n file /tmp/m.$i & done; wait' || break
done
container exec t ps -o pid,etime,stat,args | grep ssh-keygen

Result on our machine: a round below 30 (run 4), round 8 (run 5, agent holding one key) and round 1 (run 6, agent holding no key) never returned.
Inside, 14 of the 16 ssh-keygen sit in S, blocked in read on the agent socket, for as long as we let them (20 minutes).
Meanwhile container exec t echo alive still answers, so the relay as a whole is not stuck.

On the host, while they hang:

• lsof -p $(pgrep -x ssh-agent) | grep -c unix → 16: the agent holds one connection per hung client.
• netstat -f unix | awk '$3>0 || $4>0' → no unix socket with anything queued: the requests were consumed and no answer is waiting anywhere on the host.
• killall ssh-agent → every hung ssh-keygen exits at once with error fetching identities: communication with agent failed.

Negative controls

• One client against a silent agent (a nc -l -U socket that accepts and never answers): the request crosses (5 bytes reach nc), exec keeps answering, the client fails when nc dies. No hang.
• Eight clients at once against the same silent agent: exec keeps answering. But in every run one of the eight outlives nc by 40 s and more without ever seeing EOF (ssh-add -l in S inside) — a smaller instance of the same loss.
• Thirty rounds of eight parallel ssh-add -l against the real agent, in alpine, with and without a key in the agent: clean.
  Sixteen ssh-keygen -Y sign per round in the Debian image is what hangs.
• Serialising the clients (one at a time) over thirty full test-suite runs on the original workload: clean.

The terminal half, from two catches on a real machine

Same container kind, a test suite running ssh-keygen -Y sign from several threads:

• 07.09: 13 ssh-keygen blocked in read on the agent socket, the oldest three started within 60 ms of each other; three seconds after that the last byte reached the terminal master (/dev/pts/0 mtime), and zellij stayed blocked in write on it — a non-blocking write to the pty returned EAGAIN, the master side full and unread. container exec did not return either. ssh-add -l on the mac answered in 5 ms throughout. killall ssh-agent on the mac: terminal back at once, the 13 signers gone, no restart.
• 30.08: zellij in write for 159 consecutive 2-second samples; on the mac wezterm-gui in PtyFd::read, all 21 threads of container-runtime-linux in kevent.

The scripted repro has no tty attached and its terminal half never triggered; we could not make the second stage deterministic.

Where we looked in the sources

• vminitd/Sources/VminitdCore/ProcessSupervisor.swift:52-64: one thread runs epoll_wait and calls every relay handler inline; handlers are looked up by descriptor number at dispatch time.
• Sources/ContainerizationOS/Linux/Epoll.swift:121-124: descriptors are set O_NONBLOCK and registered edge-triggered.
• VsockProxy.swift:257 and :309: the client and server descriptors of each proxied connection register on that thread with [.input, .output]; transferData (:373) splices until EAGAIN via OSFile+Splice.swift:60.
• IOPair.swift:146: the terminal relay's write to the other side is a plain blocking write on the same thread.

We could not identify by reading which edge is missed; the empirical shape — live connection, EOF still delivered, data not — is what we can offer.

Problem description

m

Environment

- macOS 26.6.2, container 1.3.1 (containerization 0.42.0), guest kernel 6.18.35
- container created with `--ssh`; the host agent is Apple's `ssh-agent` (launchd) — reproduced with it holding one key and with it empty
- guest image: Debian 12 with OpenSSH 9.2p1 (reproduced there); alpine with `openssh-client` was used for the negative controls below

Code of Conduct

• I agree to follow this project's Code of Conduct