Skip to content

Fix flow-state merge freeze on repeated PROGRESS snapshots - #174

Open
rowanadair-r7 wants to merge 1 commit into
Velocidex:upstreamfrom
rowanadair-r7:ENDPOINT-20496-flow-stats-merge-freeze
Open

rowanadair-r7 wants to merge 1 commit into
Velocidex:upstreamfrom
rowanadair-r7:ENDPOINT-20496-flow-stats-merge-freeze

Conversation

@rowanadair-r7

Copy link
Copy Markdown

Problem

mergeRecords() in services/launcher/flows.go only replaces the accumulated
QueryStats when the incoming stats record represents a genuine transition
from running to a terminal state:

if len(stats_context.QueryStats) > 0 {
    if len(collection_context.QueryStats) == 0 ||
        is_running(collection_context) && !is_running(stats_context) {
        collection_context.QueryStats = stats_context.QueryStats
    }
}

Two consecutive still-running (PROGRESS) snapshots never update it. If a
flow's query never reaches a terminal state (e.g. it's stuck on a slow
upload), the collection context is permanently pinned at the very first
progress snapshot it ever received - even though the client keeps sending
fresher progress for as long as the query runs. Since UpdateFlowStats
derives TotalUploadedBytes, TotalCollectedRows, TotalLogs, State, etc.
purely from QueryStats, the whole flow view gets stuck showing stale/empty
progress and never reflects what the client has actually done.

Fix

Each FlowStats message the client sends already contains the complete,
cumulative status of every query in the flow so far (built by iterating the
full responders slice - see responder.FlowContext.getStats() upstream),
never a partial delta. So it's always safe to adopt the latest snapshot
wholesale as long as the accumulator hasn't reached a terminal state yet:

if len(stats_context.QueryStats) > 0 {
    if len(collection_context.QueryStats) == 0 ||
        is_running(collection_context) {
        collection_context.QueryStats = stats_context.QueryStats
    }
}

This still freezes correctly once the accumulator reaches a terminal state
(so a stray/out-of-order message arriving after completion doesn't reopen
it), it just stops discarding fresher in-progress snapshots.

Testing

Added TestFlowStatsProgressNotFrozen in ingestion/ingestor_test.go,
which sends two consecutive PROGRESS snapshots via
Ingestor.HandleFlowStats with increasing row/byte counts and asserts
GetFlowDetails reflects the second, fresher snapshot rather than staying
pinned at the first. Verified it fails against the old logic and passes with
the fix. Also ran the full existing suite (go test -p 1 ./...) - no
regressions; TestLauncher and TestListDirectory are unaffected since
they only exercise a single running->terminal transition, where old and new
logic agree.

mergeRecords() only replaced the accumulated QueryStats when the
incoming record was a genuine completion (running -> terminal). Two
consecutive still-running snapshots never updated it, so a flow whose
query never reaches a terminal state gets permanently pinned at the
very first progress snapshot it ever received, even though the client
keeps sending fresher progress. Each FlowStats message is already the
complete cumulative status of the flow so far (see
responder.FlowContext.getStats() upstream), so it's always safe to
adopt the latest snapshot wholesale while the flow is still running.
@CLAassistant

CLAassistant commented Sep 15, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@rowanadair-r7
rowanadair-r7 changed the base branch from master to upstream September 15, 2026 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants