Control-plane freeze #3: full-tree metadata scan of JuiceFS mount starves node (memory exhaustion + storage stall)

[bayerhazard]

Summary

Third occurrence of the node-wide control-plane freeze on Olares 1.12.6 (same host, same failure signature as reported in #3971). This time the trigger was not an LLM install/update but a full-tree metadata scan of the JuiceFS mount, which starved the entire node (etcd, rsyslog, sshd, kubelet) via memory exhaustion + storage stall.

Timeline (journal evidence)

Time	Event
~20:15–20:30	Full-tree walk over /olares/rootfs (juicefs info-style output): 696,154 files / 95,430 dirs / 718 GiB traversed via FUSE + Redis metadata
20:31:07	etcd range requests begin hanging
~20:31–20:35	System-wide I/O stall: JuiceFS chunk read from local MinIO took 4m38s (slow request ... timeout after 1m0s, cost: 4m38.4s); etcd ranges 4m39s; rsyslog file actions suspended (log writes failing)
20:35:46–48	OOM killer fires (container killed, exit 137); sshd MaxStartups throttling; k3s remotedialer i/o timeout
20:36:09	Last journal entry — total freeze. Recovery only via physical reboot

Analysis

• The metadata walk floods the FUSE daemon and Redis meta engine with lookups while holding per-inode state; combined with concurrent chunk I/O this produced minutes-long stalls across the entire disk stack.
• Memory pressure led to page-cache/writeback thrash; every userspace component sharing the node was starved simultaneously.
• The OOM kill came too late — the system was already in an unrecoverable thrash state.
• Different trigger, identical failure mode as Control-plane freeze on LLM install/update: circular JuiceFS/MinIO dependency causes storage deadlock #3971: a single shared storage stack (FUSE-JuiceFS + local MinIO on one LVM volume) with no isolation between data-plane load and control-plane services.
• Correction to Control-plane freeze on LLM install/update: circular JuiceFS/MinIO dependency causes storage deadlock #3971: the JuiceFS object backend is a local MinIO on LVM (host process), not MinIO-on-JuiceFS; the circular-dependency framing there applies only to app-level object stores running inside the mount.

Hardening recommendations

1. Bound metadata-heavy operations: run tree scans / size accounting under cgroup limits (e.g. MemoryMax) so they kill themselves instead of the node; avoid full-tree walks on live mounts during workload activity.
2. Disable writeback_cache (synchronous uploads) to remove the un-flushed queue that amplifies stall windows.
3. Add a storage-path watchdog / graceful degradation: a stalled FUSE mount should degrade affected workloads, not freeze etcd/sshd/kubelet.
4. Consider a memory ceiling for the juicefs mount client process.

Classification

Systemic weakness (missing isolation + missing degradation), not a bug in a single component.

[eball]

Hi, @bayerhazard
Thank you for the bug report.
To help us investigate further, could you share a bit more detail about the environment? Specifically:

• The typical runtime load on the control-plane machine, including memory usage and CPU utilization
• The approximate data size and file count under the JuiceFS mount directory

Additionally, could you describe the scenario in which the metadata full scan was triggered? That would help us attempt to reproduce the issue on our end and pinpoint the root cause.

Thanks again for your help!

[bayerhazard]

Thanks for following up — details below.

Environment

• Olares 1.12.6 on an "Olares One" appliance; single control-plane node (olares), one additional worker node
• Control plane hardware: Intel Core Ultra 9 275HX (24 cores), 93 GB RAM, Ubuntu 24.04, kernel 6.14
• Typical runtime load: memory used 16–30 GB steady state (63–77 GB available); load average 3–6 baseline, spikes to ~14 during LLM inference activity; k3s + llama-server are the top CPU consumers
• Storage stack is fully collapsed onto one LVM volume (1.76 TB) shared by everything:
  • JuiceFS client cache /olares/jfscache
  • Local MinIO object store /olares/data (RELEASE.2023-05-04, host process)
  • Redis 5.0.14 metadata (AOF enabled, DB1 ≈ 1.59 M keys, 233 MB on disk)
• JuiceFS client: 1.3.0+2025-07-03.30190ca
• Mount options: -o writeback_cache --entry-cache 300 --attr-cache 300 --cache-dir /olares/jfscache

Data size / file count under the mount

Captured from the very process that triggered the incident (output logged at 20:35:56):

| PATH      |   SIZE  |  DIRS |  FILES |
| /         | 718 GiB | 95430 | 696154 |
| userspace/| 683 GiB | 93717 | 669640 |

Scenario of the metadata full scan

We cannot attribute the trigger with certainty. What we know:

• syslog captured the completion output of a juicefs info-style invocation (client PID 43051) at 20:35:56 — i.e., a full-tree walk over ~700 k files had been running during the degradation window
• Active SSH sessions existed around that time from an in-cluster pod IP and a LAN host, authenticated as the same user; we could not determine whether it was a manual command (du, juicefs info) or a UI-initiated directory-size computation
• Notably, the walk completed after the OOM kill (20:35:47) and printed its table into the journal right before the last ever log lines

Suggested reproduction recipe

On a mount with similar shape (~700 GiB / ~700 k files, writeback_cache enabled, chunk I/O served by a co-located MinIO on the same disk):

1. Generate background chunk read/write traffic (or serve live workloads)
2. Run juicefs info <mountpoint> (or du -sh <mountpoint>)
3. Observe client RSS growth, slow request warnings on chunk GETs, and system-wide effects: etcd linearizable read latency (we measured 100 ms → 4 m39 s), rsyslog file-action suspension, sshd banner timeouts, eventually TaskOOM

Additional failure-signature detail (3 occurrences, identical)

Minutes before each freeze: etcd read latencies climb steadily; JuiceFS logs slow request ... timeout after 1m0s, cost: 4m38s for chunk reads from the local MinIO; rsyslog omfile actions get suspended; then a TaskOOM event fires, and the node stops logging entirely until a physical power cycle.

We're currently preparing two client-side mitigations to test: disabling writeback_cache (synchronous uploads) and capping the mount client with MemoryHigh/MemoryMax in its systemd unit so a runaway walk kills the client instead of starving the whole node. Happy to report back with results.

[chenglin01]

Hi there,

Thank you so much for providing such a comprehensive breakdown of your environment and the detailed data specs. This level of detail—especially regarding the storage stack (JuiceFS + MinIO + Redis) and the metadata scan behavior—is incredibly helpful.

Our engineering team is currently setting up a mirrored environment with a similar configuration to reproduce this issue on our end. We are actively investigating the underlying trigger for this full metadata tree traversal.

We will keep you posted as soon as we have any concrete findings or if we need any specific debug logs from your side. Thanks again for your patience and for helping us dig into this!

Cheers,
Olares Team