feat(chart): add WAL storage for alloy when eventing is enabled

[rohan2794]

• Enable alloy WAL (loki.write) with a hostPath volume on each DaemonSet node pod. WAL buffers logs to disk during Loki unavailability and auto-cleans segments after 1h (default).
• Scope alloy pod discovery to its own node via field selector.
  Add NODE_NAME downward API env var to the alloy DaemonSet so each pod
  can reference its own node name in the alloy config. Use the
  discovery.kubernetes selectors block (selectors { role = "pod"; field =
  "spec.nodeName=" + env("NODE_NAME") }) to scope each alloy instance to
  only watch pods on its own node.
  This prevents alloy daemonSet log collector
  should only watch pods on its own node, not cluster-wide.

Alloy had no WAL configured, so it used only an in-memory queue for buffering logs before shipping to Loki.

Scenario that triggered log loss:

1. Loki started but wasn't ready yet (needed "at least 2 live replicas"),
2. During that time all three alloy pods were collecting pod logs and trying to push to Loki
3. Loki responded with HTTP 500 on every push attempt
4. With no WAL, alloy's in-memory queue eventually filled up
5. Alloy on node-0 and node-1 exhausted all retries and went silent — their in-memory buffers were full and they started dropping logs
6. Only alloy on node-2 recovered when Loki finally became healthy
7. Result: pods scheduled on node-0 and node-1 had no logs in Loki

Loki collected pod logs — only from node-2. Events that existed in the NATS stream were not visible via the Loki path because the eventing-aggregator pod logs were lost as eventing-aggregator pod was scheduled on node-0.

Fix: Instead of dropping logs when the in-memory queue fills, alloy writes them to disk (/var/local/alloy/wal). When Loki recovers, alloy replays the WAL. Segments older than 1h (default value) are dropped automatically if alloy is unable to push logs to Loki.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request