Skip to content

Commit 5e26c9b

Browse files
committed
Merge commit '2ec4a3c6789b49d70ff2a37ffd7f5cf892524400'
2 parents 5d39af6 + 2ec4a3c commit 5e26c9b

4 files changed

Lines changed: 253 additions & 37 deletions

File tree

‎CONTEXT.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -181,6 +181,8 @@ Genesis supports a VSCode Remote-SSH-like workflow: a lightweight headless daemo
181181

182182
**Design constraint — multiple remote connections:** Each SSH target gets a **unique per-target node name** (`genesis_remote_<id>@127.0.0.1`, derived from the target's slugified `id`) and a per-target systemd unit (`genesis-remote-<id>`) / launchd plist label (`com.genesis.remote.<id>`). This enables **multiple simultaneous remote connections** — each target connects to a distinct BEAM node name, so `Node.connect` can address multiple hosts. The SSH tunnel uses a dynamically-assigned local port (via `find_free_port/0`) that forwards to the remote daemon's fixed port 9000. The custom `EvoGit.EpmdDist` module maps each per-target node name to its tunnel's local port via `:persistent_term` so the VM's `port_please/2` callback resolves it correctly without an external EPMD process. The remote daemon's `RELEASE_NODE` is set per-target at launch time via `--setenv=RELEASE_NODE` (systemd-run) or an `EnvironmentVariables` dict (launchd plist), and `rel/genesis_remote/env.sh.eex` respects an existing `RELEASE_NODE` env var (`export RELEASE_NODE="${RELEASE_NODE:-...}"`).
183183

184+
**SSH tunnel readiness (connect-path invariant)**: `RemoteConnection.do_connect_distributed/1` polls the local forwarded port for readiness (`wait_for_tunnel/4`, 100 ms interval, 10 s budget) BEFORE calling `Node.connect/1` — `ssh -L` can take >1.5 s to start listening (a fixed 500 ms settle sleep used to race it → instant `:econnrefused` → `{:error, {:node_connect_failed, ...}}` on fast-connecting hosts). The poll fails fast when the ssh Port dies, draining its output into the error; timeout/death → `{:error, {:tunnel_not_ready, reason}}`. `connect/1`'s GenServer call timeout is `@connect_call_timeout_ms` (wait budget + 15 s) so callers can't get a raw 5 s `:timeout`. Diagnostics: no remote-daemon log entry for a connect attempt ⇒ tunnel/port race (not a daemon problem); a `"Invalid challenge reply"` error ⇒ cookie mismatch. See `apps/evo_git/CONTEXT.md` for details.
185+
184186
**Remote asset download**: `EvoGit.RemoteBootstrap.download_url/1` is deterministic and network-free, always returning the direct `https://genesis.evox.group/dl/genesis_remote_<platform>.tar.xz` URL — a Cloudflare worker "smart download" endpoint that auto-detects whether the user is in mainland China and proxies the GitHub release asset through the Cloudflare network when needed (`curl -fL`/`wget` follow redirects). **glibc is the default and ONLY published Linux variant — asset names are NEVER suffixed** (the musl build is disabled for now, so the `_glibc` suffix, the libc-probe machinery, and `effective_libc`/`detect_libc`/`probe_libc` were all removed from `EvoGit.RemoteBootstrap` / `EvoGit.RemoteConnection`; if musl is ever revived, the suffix logic must be reintroduced deliberately). The `version` in `{:ok, url, version}` is always `"latest"` (keys the local download cache). CI uploads unversioned names (`genesis_remote_<os>_<arch>.tar.xz` — build-desktop.yml; Linux tarballs come from the dedicated `build-linux-remote` job on ubuntu-22.04, darwin/windows from their platform jobs), so no asset matching is needed. **Do NOT reintroduce asset listing/matching or any suffix check** — the desktop (`genesis_desktop_*_*.tar.gz`) and remote (`genesis_remote_*_*.tar.xz`) tarballs share the `_<os>_<arch>` suffix shape, so a bare suffix match would select the wrong asset, and the API call is unnecessary. **Format**: the remote asset is `.tar.xz` (`genesis_remote_<platform>.tar.xz`, created with `tar -cJf` in CI, downloaded/extracted with `tar -xJf` on the remote host); the desktop portable archive is `.tar.gz`. All runner OSes and remote hosts (Linux/macOS) have xz-capable tar, so no fallback exists — xz failure fails loudly.
185187

186188
### Desktop App Build Pipeline

‎apps/evo_git/CONTEXT.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,10 @@ The SSH remote-bootstrap release asset is a **`.tar.xz`** archive: `genesis_remo
7373

7474
**Linux asset variants:** glibc is the default and only published Linux variant — remote tarball names are NEVER suffixed (`genesis_remote_linux_x64.tar.xz`, `genesis_remote_linux_arm64.tar.xz`). The musl build is disabled for now. The previous libc-variant machinery was removed: `detect_libc/1` (ldd parser), `probe_libc/1` (SSH `ldd --version` probe), the private `effective_libc/2` normalization, the `_glibc` suffix in asset names/URLs/cache paths, and the `libc` params threaded through `resolve_platform` → `download_tarball` → `download_url`/`cache_path` (`resolve_platform/2` now returns `{:ok, daemon_os, platform}`). Downloads always resolve the default unsuffixed glibc asset. On NixOS hosts the glibc-only Linux asset runs via on-the-fly patchelf patching during bootstrap (`EvoGit.RemoteBootstrap.nixos_patch_script/1` + the `:patching_binaries` stage) — no new asset variants; glibc stays the only published Linux variant.
7575

76+
### SSH Tunnel Readiness (connect-path invariant)
77+
78+
`EvoGit.RemoteConnection.do_connect_distributed/1` must NOT call `Node.connect/1` before the SSH tunnel's local forwarded port actually accepts TCP connections. An `ssh -L <local_port>:127.0.0.1:<dist_port> -N <target>` forward can take well over a second to start listening on the local port (measured ~1.5s), and a too-early `Node.connect` fails instantly with econnrefused → `{:error, {:node_connect_failed, ...}}` with NO remote-side log entry (the dist handshake never reaches the daemon — do not hunt for cookie/EpmdDist causes when the remote log shows nothing). The connect path therefore polls local-port TCP readiness via `wait_for_tunnel/4` (`@tunnel_poll_interval_ms` 100ms, `@tunnel_wait_timeout_ms` 10s budget), fails fast when the ssh Port dies (draining its output into the error for diagnosability), and returns `{:error, {:tunnel_not_ready, reason}, state}` on timeout/death. `connect/1`'s GenServer call timeout is `@connect_call_timeout_ms` (wait budget + 15s) so callers can't exit with a raw `:timeout` while the handler is legitimately waiting. `wait_for_tunnel/4` is `@doc false` public solely so tests can drive it with tiny timeouts/poll intervals (see `test/evo_git/remote_connection_test.exs`). Remote-side rejection ("Invalid challenge reply" in the daemon journal) means a COOKIE mismatch — a different failure mode with a visible remote log entry.
79+
7680
## Custom Agents & Model Selection
7781

7882
User-defined agents and per-agent model selection live in ONE file: `<config_dir>/agents.toml` (same dir as config.toml, next to `remote_connections.toml`). Per-node semantics: agents.toml is read from the config dir of the node executing the agent — local node for local tasks, remote host for tasks on an SSH remote node (the RPC edits the file on the node whose config is being edited). The scheduler has NO cached copy; `Dispatch.register_agent/7` calls `EvoGit.CustomAgents`/`EvoGit.CustomAgents.ModelSelector` directly per spawn (both are stat-validated-cache backed, so per-spawn cost is one `File.stat` + one `:persistent_term.get`).

‎apps/evo_git/lib/evo_git/remote_connection.ex‎

Lines changed: 133 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,17 @@ defmodule EvoGit.RemoteConnection do
8282
@supervisor EvoGit.RemoteConnection.Supervisor
8383
@pubsub_topic "remote_connections"
8484
@heartbeat_interval_ms 10_000
85-
@tunnel_settle_ms 500
85+
# Wait budget for the SSH tunnel to start forwarding the local port before
86+
# Node.connect. Measured need is ~1.5s (an `ssh -L` forward can take that
87+
# long to start listening on the local port); a fixed 500ms sleep made
88+
# Node.connect fail instantly with econnrefused on slower tunnels.
89+
@tunnel_wait_timeout_ms 10_000
90+
@tunnel_poll_interval_ms 100
91+
# The :connect handler can legitimately block for up to @tunnel_wait_timeout_ms
92+
# while the tunnel becomes ready, so the GenServer.call in connect/1 must
93+
# allow that budget plus headroom (a default 5s call timeout would exit with
94+
# :timeout while the handler is still waiting for the tunnel).
95+
@connect_call_timeout_ms @tunnel_wait_timeout_ms + 15_000
8696
@launch_receive_timeout_ms 5_000
8797
# 900s — bootstrap now stages the release either by uploading a local tarball
8898
# (scp) OR by probing the remote platform and downloading the release tarball
@@ -192,7 +202,7 @@ defmodule EvoGit.RemoteConnection do
192202
def connect(target_id) do
193203
with {:ok, target} <- fetch_target(target_id),
194204
{:ok, pid} <- ensure_started(target) do
195-
GenServer.call(pid, :connect)
205+
GenServer.call(pid, :connect, @connect_call_timeout_ms)
196206
end
197207
end
198208

@@ -423,42 +433,54 @@ defmodule EvoGit.RemoteConnection do
423433
cmd = build_tunnel_command(target, local_port, remote_port)
424434
port = Port.open({:spawn, cmd}, [:binary, :exit_status, :stream])
425435

426-
# Give the SSH tunnel time to establish.
427-
Process.sleep(@tunnel_settle_ms)
428-
429-
# Register the SSH tunnel's local port with our EPMD-less module so
430-
# that when Node.connect asks port_please for this node name, EpmdDist
431-
# returns local_port (which the SSH tunnel forwards to remote_port 9000).
432-
node_name = remote_node_name(target)
433-
EvoGit.EpmdDist.register_target(node_name, local_port)
434-
435-
# The node name is just name@host — NO port suffix. The port is
436-
# resolved by EpmdDist.port_please/2 via the registration above.
437-
remote_node = String.to_atom(node_name)
438-
439-
case Node.connect(remote_node) do
440-
true ->
441-
ref = schedule_heartbeat()
442-
443-
new_state = %__MODULE__{
444-
connecting
445-
| phase: :connected,
446-
ssh_tunnel_port: port,
447-
tunnel_local_port: local_port,
448-
node: Atom.to_string(remote_node),
449-
heartbeat_ref: ref,
450-
last_error: nil
451-
}
452-
453-
broadcast_status(state.target, new_state)
454-
{:ok, new_state}
455-
456-
result when result in [false, :ignored] ->
457-
EvoGit.EpmdDist.unregister_target(node_name)
436+
# Give the SSH tunnel time to establish: poll the local forwarded port
437+
# for TCP readiness (bounded) instead of a fixed sleep — `ssh -L` can
438+
# take well over a second to start listening, and a too-early
439+
# Node.connect fails instantly with econnrefused.
440+
case wait_for_tunnel(port, local_port, @tunnel_wait_timeout_ms,
441+
poll_interval: @tunnel_poll_interval_ms
442+
) do
443+
:ok ->
444+
# Register the SSH tunnel's local port with our EPMD-less module so
445+
# that when Node.connect asks port_please for this node name, EpmdDist
446+
# returns local_port (which the SSH tunnel forwards to remote_port 9000).
447+
node_name = remote_node_name(target)
448+
EvoGit.EpmdDist.register_target(node_name, local_port)
449+
450+
# The node name is just name@host — NO port suffix. The port is
451+
# resolved by EpmdDist.port_please/2 via the registration above.
452+
remote_node = String.to_atom(node_name)
453+
454+
case Node.connect(remote_node) do
455+
true ->
456+
ref = schedule_heartbeat()
457+
458+
new_state = %__MODULE__{
459+
connecting
460+
| phase: :connected,
461+
ssh_tunnel_port: port,
462+
tunnel_local_port: local_port,
463+
node: Atom.to_string(remote_node),
464+
heartbeat_ref: ref,
465+
last_error: nil
466+
}
467+
468+
broadcast_status(state.target, new_state)
469+
{:ok, new_state}
470+
471+
result when result in [false, :ignored] ->
472+
EvoGit.EpmdDist.unregister_target(node_name)
473+
close_port(port)
474+
reason = {:node_connect_failed, inspect(remote_node)}
475+
new_state = %{connecting | phase: :error, last_error: format_error(reason)}
476+
{:error, reason, new_state}
477+
end
478+
479+
{:error, reason} ->
458480
close_port(port)
459-
reason = {:node_connect_failed, inspect(remote_node)}
460-
new_state = %{connecting | phase: :error, last_error: format_error(reason)}
461-
{:error, reason, new_state}
481+
error = {:tunnel_not_ready, reason}
482+
new_state = %{connecting | phase: :error, last_error: format_error(error)}
483+
{:error, error, new_state}
462484
end
463485

464486
{:error, reason} ->
@@ -1348,6 +1370,80 @@ defmodule EvoGit.RemoteConnection do
13481370
end
13491371
end
13501372

1373+
# Waits (bounded) for the SSH tunnel's local forwarded port to accept TCP
1374+
# connections, so Node.connect doesn't race the `ssh -L` startup.
1375+
#
1376+
# Polls `local_port` every `poll_interval` ms (opts, default 100) until a
1377+
# TCP connect succeeds — the probe socket is closed immediately, which the
1378+
# `ssh -N` forward handles gracefully. Fails fast when the tunnel Port
1379+
# dies: `Port.info/1` returning nil (works regardless of the caller's
1380+
# trap_exit flag), a received `{:exit_status, status}`, or a received
1381+
# `{:EXIT, port, reason}` (only delivered to trap_exit processes). Port
1382+
# output (`{port, {:data, data}}`) is drained into the error reason so ssh
1383+
# stderr is available for diagnosability.
1384+
#
1385+
# Returns `:ok` | `{:error, reason}`:
1386+
#
1387+
# * `{:ssh_exited, status_or_reason, ssh_output}` — tunnel port died early
1388+
# * `{:timeout, ssh_output}` — budget exhausted, port still alive
1389+
#
1390+
# `@doc false`: public only so tests can drive it directly with tiny
1391+
# timeouts; it is an implementation detail of `do_connect_distributed/1`.
1392+
@doc false
1393+
@spec wait_for_tunnel(port(), :inet.port_number(), non_neg_integer(), keyword()) ::
1394+
:ok | {:error, term()}
1395+
def wait_for_tunnel(port, local_port, timeout_ms, opts \\ []) do
1396+
poll_interval = Keyword.get(opts, :poll_interval, 100)
1397+
wait_for_tunnel(port, local_port, timeout_ms, poll_interval, "")
1398+
end
1399+
1400+
defp wait_for_tunnel(port, local_port, remaining_ms, poll_interval, ssh_output) do
1401+
if Port.info(port) == nil do
1402+
{:error, {:ssh_exited, :port_info_nil, ssh_output}}
1403+
else
1404+
case tcp_ready?(local_port) do
1405+
:ok ->
1406+
:ok
1407+
1408+
:not_ready ->
1409+
wait = min(poll_interval, remaining_ms)
1410+
1411+
receive do
1412+
{^port, {:data, data}} ->
1413+
wait_for_tunnel(port, local_port, remaining_ms, poll_interval, ssh_output <> data)
1414+
1415+
{^port, {:exit_status, status}} ->
1416+
{:error, {:ssh_exited, status, ssh_output}}
1417+
1418+
{:EXIT, ^port, reason} ->
1419+
{:error, {:ssh_exited, reason, ssh_output}}
1420+
after
1421+
wait ->
1422+
remaining = remaining_ms - wait
1423+
1424+
if remaining <= 0 do
1425+
{:error, {:timeout, ssh_output}}
1426+
else
1427+
wait_for_tunnel(port, local_port, remaining, poll_interval, ssh_output)
1428+
end
1429+
end
1430+
end
1431+
end
1432+
end
1433+
1434+
defp tcp_ready?(local_port) do
1435+
case :gen_tcp.connect({127, 0, 0, 1}, local_port, [:inet, :binary, {:active, false}], 250) do
1436+
{:ok, socket} ->
1437+
# Readiness probe — close immediately, we only need to know the
1438+
# forwarded port is accepting connections.
1439+
:gen_tcp.close(socket)
1440+
:ok
1441+
1442+
{:error, _reason} ->
1443+
:not_ready
1444+
end
1445+
end
1446+
13511447
# ── Helpers ────────────────────────────────────────────────────────
13521448

13531449
defp remote_node_name(target) do

0 commit comments

Comments
 (0)