[Bug]: VM-to-VM TCP frames arrive above the guest MTU without GSO metadata, so a receiving VM cannot forward them (uploads stall unless the sender disables TSO)

[saiyam1814]

I have done the following

• I have searched the existing issues
• If possible, I've reproduced the issue using the 'main' branch of this project

Steps to reproduce

Two ordinary containers on the default network, no Kubernetes involved. The receiver runs a small HTTP PUT sink inside a network namespace behind a veth, reachable through an iptables DNAT (the shape of a NodePort into a pod, or a docker-in-container port publish); the sender uploads 1 MiB.

# receiver
container run -d --name tso-rx --cap-add NET_ADMIN --cap-add SYS_ADMIN --cap-add NET_RAW \
  --read-only-path NONE python:3.12-alpine sleep infinity
container exec tso-rx sh -euc '
apk add --no-cache iptables iptables-legacy iproute2 >/dev/null
ip netns add srv; ip link add veth0 type veth peer name veth1; ip link set veth1 netns srv
ip addr add 10.9.0.1/24 dev veth0; ip link set veth0 up
ip netns exec srv ip addr add 10.9.0.2/24 dev veth1; ip netns exec srv ip link set veth1 up; ip netns exec srv ip link set lo up
ip netns exec srv ip route add default via 10.9.0.1
sysctl -w net.ipv4.ip_forward=1
cat > /sink.py <<"PY"
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import sys
class H(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"
    def do_PUT(self):
        n = int(self.headers["Content-Length"]); got = len(self.rfile.read(n)); b = str(got).encode()
        self.send_response(200); self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)
ThreadingHTTPServer(("0.0.0.0", int(sys.argv[1])), H).serve_forever()
PY
nohup ip netns exec srv python3 /sink.py 8080 >/dev/null 2>&1 &
nohup python3 /sink.py 8081 >/dev/null 2>&1 &
sleep 1
iptables-legacy -t nat -A PREROUTING -p tcp --dport 30080 -j DNAT --to-destination 10.9.0.2:8080'
RX=$(container ls | awk '/tso-rx/ {print $6}' | cut -d/ -f1)
container exec tso-rx iptables-legacy -t nat -A PREROUTING -p tcp --dport 30081 -j DNAT --to-destination $RX:8081

# sender
container run -d --name tso-tx --cap-add NET_ADMIN alpine:3.20 sleep infinity
container exec tso-tx sh -euc "apk add --no-cache curl ethtool >/dev/null; head -c 1048576 /dev/urandom > /big.bin; ip route add 10.9.0.0/24 via $RX dev eth0"
put() { container exec tso-tx curl -s -o /dev/null --max-time 15 -T /big.bin -w "$1  http=%{http_code} sent=%{size_upload} time=%{time_total}s\n" "$2"; }
put "A DNAT+forward into veth " http://$RX:30080/
put "B DNAT, local delivery   " http://$RX:30081/
put "C forward into veth, no NAT" http://10.9.0.2:8080/
put "D direct                 " http://$RX:8081/
container exec tso-tx ethtool -K eth0 tso off
put "G as A, sender tso off   " http://$RX:30080/

--read-only-path NONE is needed on 1.2.1+ because /proc/sys is read-only by default (#2041); on 1.0.0, where this was measured, the flag does not exist and is not needed. iptables-legacy because the default node kernel has no nf_tables; path C needs no iptables at all.

Problem description

Three runs each, sender TSO on unless noted (container 1.0.0, macOS 26.2, default kernel 6.12.28):

Path	Result
A. VM -> DNAT + forward into veth (:30080)	stalls after 395 / 459 / 524 kB, 15 s timeout
B. VM -> DNAT only, local delivery (:30081 -> :8081)	200, 1 MiB in 8-31 ms
C. VM -> forward into veth, no NAT (10.9.0.2:8080)	stalls after 491 / 427 / 524 kB, timeout
D. VM -> direct (:8081)	200, 1 MiB in 5-21 ms
E. As A, 100 KiB body	1 timeout, then 8.8 s and 4.0 s for 100 KiB
F. As A, iptables -t mangle TCPMSS clamp 1220 on the receiver's FORWARD chain	stalls after 304 / 304 / 386 kB
G. As A, sender ethtool -K eth0 tso off	200, 1 MiB in 16-31 ms
H. As A, receiver tso/gso/gro/tx off, sender TSO on	stalls
I. As A, sender gso off only, TSO on	stalls
macOS host -> :8081, :30080, :30081 (once each)	200, 1 MiB in 9-13 ms

B versus C shows NAT is not the trigger; F shows MSS is not the trigger; H rules out receiver-side GRO; the host sending through the identical forward path is fine. The only lever is segmentation offload on the sending VM.

What the receiver sees on path C (tcpdump on its eth0, MTU 1280, and inside the namespace on veth1, MTU 1500; nstat for one attempt):

eth0:   IP ... length 6192  seq 0:6140        (5 coalesced 1228-byte segments in one packet, DF set)
        IP ... length 6192  seq 6140:12280
        IP ... length 4964 / 2508 / 1280      (4, 2 and 1 segments per packet)
veth1:  only the length-1280 packets arrive; the larger ones never do
nstat:  IpForwDatagrams 1025   IpFragFails 328   IcmpOutDestUnreachs 328   TcpExtTCPOFOQueue 77 (in the namespace)

So the sending VM's TSO super-frames reach the receiving VM as single IP packets of up to 6192 bytes on a 1280-byte interface. A local socket accepts them (paths B, D). Forwarding them into the veth must fit the egress MTU, DF is set, so ip_forward drops them with IpFragFails and an ICMP fragmentation-needed per packet; only the un-coalesced segments get through, and TCP crawls on those until it gives up. That is the behaviour of an skb with no GSO metadata (gso_size == 0); a GSO skb would have been re-segmented on egress instead. gso_size is not visible in a capture, so this is inference from the counters, supported by two controls: with sender TSO off the same stream arrives as 1228-byte segments, the receiver's own GRO builds real GSO skbs (6192-byte packets then appear on veth1 too) and the PUT completes in about 15 ms; and raising the veth MTU changes the outcome exactly as an untagged oversized packet would (MTU 1280 and 1500 both stall with IcmpOutDestUnreachs equal to IpFragFails, 410 and 317; MTU 65535 completes 1 MiB in 9 ms with zero fragmentation failures). Guest-side evidence cannot separate "host never set the virtio-net GSO header" from a guest negotiation problem, but the receiver's virtio-net did negotiate VIRTIO_NET_F_GUEST_TSO4/6 (features bits 7, 8, 11, 12 set) and reports rx-gro-hw: on [fixed], so it would have honoured the metadata had the frames carried any.

Expected: a TSO-enabled guest should be able to send a multi-hundred-kB TCP stream to another guest regardless of what the receiver does with it after L3 forwarding, as it can when the receiver delivers it locally, and as the macOS host can through the same DNAT and veth.

Impact: any NodePort or LoadBalancer on a container-backed Kubernetes node, and any docker-in-container port publish, stalls for uploads from other VMs on the same Mac. Guest-side workarounds exist (kiac turns TSO off on every node NIC, or terminates TCP on the receiving VM before the veth), but none is something a CNI or an application does by default; the clean fix is for forwarded VM-to-VM frames to arrive either segmented to the guest MTU or with GSO metadata the guest can honour. First seen through Kubernetes on 1.0.0 and 1.1.0 (saiyam1814/kiac#8, reported by @ldub, confirmed on a second machine); reproduced above with no Kubernetes involved.

Environment

• OS: macOS 26.2 (25C56), Apple silicon
• Xcode: Command Line Tools only
• Container: container CLI version 1.0.0 (build: release, commit: ee848e3), default kernel 6.12.28. Four releases behind 1.3.1 and not yet re-run there; the script above takes about two minutes if a maintainer wants to check it on current.

Code of Conduct

• I agree to follow this project's Code of Conduct

[saiyam1814]

Re-ran the same script on the current release: container CLI 1.3.1, macOS 26.2, default kernel 6.12.28, receiver started with --read-only-path NONE as noted. Same pattern as on 1.0.0 (A, C, E and F stall; B, D, G and the host pass), three runs each, 1 MiB body, sender TSO on unless noted:

Path	1.3.1 result
A. DNAT + forward into veth	stalls after 360 / 393 / 172 kB, 15 s timeout
B. DNAT only, local delivery	200, 8.1-11.9 ms
C. forward into veth, no NAT	stalls after 435 / 491 / 204 kB
D. direct	200, 6.6-11 ms
E. as A, 100 KiB body	3.6 s, 3.2 s, then one timeout
F. as A, receiver MSS clamp 1220	stalls after 380 / 141 / 393 kB
G. as A, sender tso off	200, 14.9-28.8 ms
host -> :8081 / :30080 / :30081	200, 9-16 ms

So this is not specific to 1.0.0; it is present in 1.3.1 with the default kernel.

[saiyam1814]

One correction to the 1.3.1 run above: that kernel was 6.12.28 left over from a 1.0.0 install (upgrades keep the existing kernel, #905), not the kernel 1.3.1 recommends. The numbers stand.

Re-ran on 1.4.1 with the recommended vmlinux-6.18.35-197-debug. Same picture. A, C and F stall after 141 to 524 kB. B, D, G and the host paths move 1 MiB in 5 to 26 ms; E takes 3.7 s, 3.7 s, then times out. So this still reproduces on 1.4.1 with the recommended kernel.