Skip to content

Use kernel event queues for socket monitoring - #23

Open
colemancda wants to merge 34 commits into
mainfrom
feature/kernel-event
Open

Use kernel event queues for socket monitoring#23
colemancda wants to merge 34 commits into
mainfrom
feature/kernel-event

Conversation

@colemancda

Copy link
Copy Markdown
Member

Replaces the global poll(2) set with per-platform kernel event queues, behind a seam that keeps poll as the fallback for platforms without one.

Event queue seam

EventQueue is an internal protocol with three implementations, selected at compile time:

Platform Backend
Linux, Android epoll(7)
Darwin kqueue(2)
everything else poll(2)

FileEvents stays the only vocabulary above the seam, so no EPOLLIN or EVFILT_READ leaks into the manager. wait yields a borrowed buffer instead of returning an array, so a tick allocates nothing. Both kernel backends are level triggered to match poll semantics exactly, and each carries a wake channel (eventfd on Linux, EVFILT_USER on Darwin) for a future blocking driver.

kqueue reports read and write as separate filters, so entries are coalesced per descriptor before being handed up, otherwise the manager would resume the same continuation twice.

Write readiness is now registered on demand

A connected socket is almost always writable, so a standing EPOLLOUT/EVFILT_WRITE registration made every wait return every socket on every tick, which is most of what the old poll loop was doing. Sockets now register for read, error and hangup only, and write readiness is added while a write is pending and dropped once nothing is waiting on it.

Idle CPU over 5 seconds with 400 connected sockets:

CPU
before 0.176 s
after 0.004 s

Behavior change: Socket.event no longer emits spurious .write notifications for idle sockets. A socket that only reads never sees .write, and a socket that writes sees it once for the pending write rather than repeatedly. Code that treated .write as a periodic heartbeat will notice.

Descriptor ownership

poll reported POLLNVAL for a descriptor closed behind the manager's back, so its entry was cleaned up promptly. Kernel queues drop closed descriptors silently, which let stale entries accumulate until a late remove closed a descriptor number the kernel had already handed to a new socket. This reproducibly crashed the suite on Linux.

The manager no longer closes descriptors it did not open. On hangup or error it deregisters and finishes the event stream but leaves the descriptor open, recording it so its owner can still close it exactly once. Because the descriptor stays open, the kernel cannot recycle the number, which removes the race. No public API change.

Syscall mocking

The ENABLE_MOCKING hooks were already scattered through Syscalls.swift, but the driver types they referenced never existed, so that path could not compile. Adds a thread local MockingDriver with a syscall Trace and ForceErrno, modeled on swift-system, fixes four hooks that could not type check, and enables the flag for debug builds.

Tests

  • EventQueueTests runs one conformance suite against every backend, so the poll implementation stays correct on platforms where it is no longer the default. Covers write then read readiness, level triggered persistence, interest mask changes, end of file, and deregistration.
  • IdleTests asserts idle CPU stays under 20µs per socket second. Restoring the standing write registration makes it fail at 87µs against 3µs, so it is a real guard rather than decoration. A companion test asserts idle sockets report no events at all, which is deterministic and cannot flake.
  • MockingTests covers tracing, forced errno, counted errno and scoping.

CI now runs tests serially, since these share the process file descriptor table and the idle benchmark measures CPU for the whole process. The Linux job previously only built; it now runs the suite too, which is the first coverage the epoll path gets.

Verification

macOS is green across repeated full runs, apart from a pre-existing No route to host failure in the UDP test that also fails on main. On Linux the conformance and mocking suites pass under swift:6.0, but the full suite could not be verified locally: the socket tests are unreliable in local containers on main as well, hanging or failing on loopback. The newly enabled Linux job is the real check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant