[Bug] An Activity that never returns prevents worker.run() from ever settling after a fatal error

[TastyPi]

What are you really trying to do?

Running TypeScript Workers in Kubernetes. When a Worker reaches a state where it can no longer process work, we want the process to exit so the orchestrator replaces the pod. We rely on worker.run() surfacing fatal errors for that — an unhandled rejection from it is what terminates our process.

Describe the bug

If any Activity is in flight and does not return, worker.run() never settles after a fatal error — it neither resolves nor rejects — and the process runs indefinitely. Nothing is logged after Initiating Worker shutdown, so the application cannot detect the condition either.

run() awaits merge(workflow$, activity$, nexus$) completing (worker/lib/worker.js, runInternal). The captured fatalError is only rethrown from the tap when that stream completes or errors, so while any Activity remains outstanding the promise stays pending and no Worker failed is logged. The only escape is forceShutdown$(), which returns EMPTY when shutdownForceTimeMs == null (worker/lib/worker.js:636) — the default, since #1072 made shutdown non-forceful unless shutdownForceTime is set explicitly.

Cancellation does not rescue this. shutdownGraceTime defaults to 0, so cancellation is requested immediately and is correctly delivered — we verified an Activity observing CancelledFailure: WORKER_SHUTDOWN via listener, polled flag and the cancelled promise. But cancellation is only a request: an Activity doing work that cannot be interrupted — a tight loop, a synchronous or non-abortable I/O call, a third-party SDK with no abort support — simply continues, and the drain waits on it forever.

This appears to be the state #1539 set out to eliminate, quoting its own description:

this could possibly result in various "zombie states", e.g. the Worker's run promise not resolving/failing even though the Worker has reached FAILED state

Worker.runUntil() was given a bound for exactly this — promiseCompletionTimeout, defaulting to 0, on the reasoning that "we shouldn't wait". Bare run() was not, so the same hang remains reachable there.

It also falls short of the intent stated when closing #1536, which described this same symptom (ERR_WORKER_OUT_OF_MEMORY, no further progress, "the Worker process is still running", persisting until restart):

the correct behavior would be for the Worker to initiate shutdown ... Should probably print a clear CRITICAL level message to the log, and terminate the process ASAP.

Production impact. Two Workers stopped processing and stayed alive for ~39 hours, recovering only when an unrelated deploy replaced the pods. Nothing in the process could react: no exception was ever thrown, so our process.on('uncaughtException') handler — which would have exited — never fired. The OOM that started it was #2227; the 39 hours are down to this.

Worth noting the two failure modes are separable, and #2227's own reproduction shows the benign one: its output ends with Worker failed and state FAILED, i.e. run() rejected and the process could exit. That repro has no long-running Activity in flight, so the drain completes immediately. The OOM alone does not hang the Worker — an Activity that cannot complete is the additional ingredient.

Minimal Reproduction

An Activity that never returns is sufficient, and deliberately one that ignores cancellation so the reproduction does not depend on any cancellation behaviour:

const worker = await Worker.create({
  connection,
  taskQueue: 'repro',
  workflowsPath: require.resolve('./workflows'),
  activities: {
    hang: () => new Promise<never>(() => {}), // never settles, ignores cancellation
  },
});

await worker.run();
console.log('never reached');

The Workflow calls hang() and retains enough memory that the sandbox thread OOMs while the Activity is outstanding. #2227 links a ready-made reproduction of the OOM itself (millerick/temporal-worker-crash-reproduction) if that is a convenient starting point — adding the hanging Activity to it is the whole delta.

Observed:

[ERROR] Workflow Worker Thread failed: Error [ERR_WORKER_OUT_OF_MEMORY]: ...
[ERROR] An unexpected error occurred while processing Workflow Activation. Initiating Worker shutdown.
[INFO]  Worker state changed { state: 'STOPPING' }
[INFO]  Worker state changed { state: 'DRAINING' }

…then nothing further. No Worker failed, no transition to FAILED, run() still pending, and the process alive with its event loop still running (we confirmed with a 2-second heartbeat timer that kept ticking). Setting shutdownForceTime: '20s' flips it: Worker failed is logged at the deadline and the process exits.

Environment/Versions

• OS and processor: Linux x86_64, containerised
• Temporal Version: Temporal Cloud (exact server build not verifiable from our side); SDK @temporalio/worker 1.21.1 (current latest)
• Are you using Docker or Kubernetes or building Temporal from source? Kubernetes (EKS), official SDK release from npm
• Node 24

Additional context

Suggested remedies, in rough order of how closely they match the intent stated in #1536:

1. Treat Workflow sandbox thread death as unrecoverable and skip the graceful drain — no Workflow Task can ever be processed again, so there is little to drain for.
2. Apply a default shutdownForceTime on the unexpected-error path only, leaving user-initiated shutdown() unchanged.
3. Guarantee run() always settles after a fatal error even if in-flight work is abandoned, mirroring the promiseCompletionTimeout: 0 decision made for runUntil in #1539.
4. Failing those, log at CRITICAL when a fatal error has been captured but the drain has not completed after some interval, so the condition is at least observable from outside.

Related:

• #2265 — NativeConnection.withAbortSignal never cancels requests, so Activity cancellation cannot abort getClient() calls. That is one concrete way to end up with an Activity that will not return. The two are independent though: fixing that still leaves any genuinely non-interruptible Activity able to hang the drain, which is what this report is about.
• temporalio/sdk-rust#1297 — same user-visible symptom (Worker.run() never returns) from a different cause (activity poll not resolving rather than an activity body blocking).
• #1739 / PR #2264 — giving Activities a shutdown signal. Helps well-behaved Activities cooperate, but cannot help one that does not.

We are adopting shutdownForceTime ourselves as a mitigation; filing because the default behaviour turned a recoverable crash into a 39-hour outage, and because the surrounding history suggests this was not the intended outcome.