[Bug]: Actor's worker IP is not revalidated after a node reboot — stays RUNNING, unreachable

[thpham]

Title: [Bug]: Actor's worker IP is not revalidated after a node reboot — stays RUNNING, unreachable

Relatives: #1526 (worker pod disappears → CRASHED, unrecoverable), #1665/#1518/#50 (stuck in a transitional state after a failed lifecycle op). This is the step before either: the actor never leaves RUNNING, silently pointing at a dead address.

What happened

After a full host reboot, worker pods come back Running with the same name but a new pod IP (fresh network sandbox). The actor controller never re-resolves it: kubectl ate get actor / ax describe task keep reporting RUNNING/Ready at the old IP. First sign of trouble is a request failing:

upstream reset: ... delayed connect error: No route to host

(also seen via ax ssh: rpc error: code = Unavailable ... remote connection failure)

Trying to fix it by hand makes it worse — suspend actor fails (runsc checkpoint: exit status 128) and strands the actor in SUSPENDING with no way out (#1665/#50). Only recovery: delete and recreate.

Repro

1. Single-node k3s, --kind profile, WorkerPool with 2 replicas.
2. Create an actor (or an AX Task — same mechanism), confirm it's reachable, note its worker IP.
3. Reboot the host.
4. Worker pod: same name, new IP (kubectl get pod <worker> -o wide before/after). Actor: still RUNNING at the old IP → unreachable, and suspend strands it in SUSPENDING.

Reproduced identically for a plain counter demo actor and for an AX Task.

Expected

The controller should revalidate/re-resolve a RUNNING actor's worker IP (or treat a recreated sandbox like a disappeared worker, #1526) — not report healthy indefinitely with no health signal.

Environment

Substrate 672533541dbf (main), --kind profile, single-node k3s 1.36.4, also reproduced through AX v0.3.0.

[BenTheElder]

If the node is rebooted under the actor that actor is no longer running and we need to treat is as crashed. That's a bug for sure.

Actors / their sandboxes are interior processes that won't automatically be recreated (and shouldn't as they'd have lost memory), but should be possible to resume from the last snapshot, so ~#1526 / #1675 .

The IP part is a red-herring.

[thpham]

Thanks — agreed, the IP change is just a symptom of the sandbox being gone; the missing crash detection is the real issue.

RevertActor (#1675) looks like exactly the recovery path we need. We're pinned to 672533541dbf (2026-09-11 — not 09-22 as I wrote above, sorry), which predates it, because that's what google/ax v0.3.0 requires. We'll pick it up once AX bumps.

[sfunkenhauser]

We mark an actor CRASHED if the pod gets deleted, which is what happens during a normal eviction flow. I'm guessing eviction didn't happen (Because the reboot was sudden?), the container was terminated and restarted by the ReplicaSet controller. We need to mark an actor as CRASHED if we notice the ateom has terminated and restarted.

[thpham]

Reproduced it on purpose to check the eviction theory — sfunkenhauser is right, and there's a second half.

Single-node k3s, counter demo. Actor A suspended before the reboot, actor B left RUNNING. Plain systemctl reboot, no drain.

After reboot:

• Worker pods: same names, RESTARTS 1, new IPs — no eviction, containers restarted in place.
• B: still RUNNING at its old IP, never marked CRASHED.
• Every worker record keeps its pre-reboot IP, so even A (cleanly SUSPENDED) resumes "successfully" onto a worker nothing can route to. ate-controller notices, but only logs it on every resync:

  Syncer: registered worker IP disagrees with its pod  registered=10.42.0.179 pod_ip=10.42.0.198
  

  That's the TODO: I don't think this is possible branch in cmd/atecontroller/internal/workersync/syncer.go — here's a deterministic repro. Since ip is immutable on a Worker, handling that mismatch like a pod deletion (drop the worker, CRASH its actor) would cover both halves.

Workaround that works today: suspend actors → reboot → delete the worker pods (fresh pods re-register with correct IPs) → resume. A came back with its state intact (counter 3 → 4); B went CRASHED as expected.

[sfunkenhauser]

That's a good point, there are 2 separate bugs here.

1. Worker IPs are never updated, which results in new actors being unreachable after they are scheduled. When an actor is assigned to a worker, the IP is copied to the actor object and used by atenet to route requests to the correct worker (ingress.go#123).

2. When ateom restarts, any running actors aren't restarted with it. These actors should be marked CRASHED. This is being tracked by [Bug]: An ateom that restarts strands all running actors #1882