What is the behavior when using the default overflow_action=throw_exception

[sean-scott-lr]

I looked in the code and it does not appear that this plugin handles the BufferOverflowException like "in_tail" does. What is the behavior if this occurs? What is the recommended overflow_action ?

[Watson1978]

Sorry for the very late reply, and thanks for raising this.

You are right that in_windows_eventlog2 does not handle Fluent::Plugin::Buffer::BufferOverflowError, and the resulting behavior is arguably worse than a plain drop.

What happens with the default overflow_action throw_exception

Reading is driven by one timer per channel, and the emit happens here:

def on_notify_hash(ch, subscribe)
  es = Fluent::MultiEventStream.new
  begin
    subscribe.each do |record, message, string_inserts|
      # ...
    end
    router.emit_stream(@tag, es)
    @bookmarks_storage.put(ch, subscribe.bookmark)
    log.trace "..."
  rescue Winevt::EventLog::Query::Error => e
    # ...
  end
end

Only Winevt::EventLog::Query::Error is rescued, so a BufferOverflowError propagates all the way out:

1. EventRouter#emit_stream hands it to RootAgent#handle_emits_error, which logs emit transaction failed: and re-raises it (it is only swallowed when a <label @ERROR> section exists).
2. The exception escapes on_notify_hash / on_notify_xml.
3. Fluent::PluginHelper::Timer::TimerWatcher#on_timer catches it, logs Unexpected error raised. Stopping the timer., and calls detach.

So the practical outcome is not just a dropped batch: the timer for that channel is detached, and that channel stops being collected until Fluentd is restarted. The one exception is when refresh_subscription_interval is set, because refresh_subscriptions re-creates the subscriptions and their timers, which effectively recovers the stopped channel.

On the positive side, @bookmarks_storage.put runs only after a successful emit, so the bookmark is not advanced for the failed batch and a restart resumes from the last successfully emitted position (the events are re-read as long as they are still present in the channel's .evtx). This is the key difference from in_tail, which returns false on BufferOverflowError and simply retries the same position on the next read, keeping the process running.

Recommended overflow_action

The first thing to do is to remove the overflow itself: size the buffer appropriately (total_limit_size, chunk_limit_size, flush_thread_count) and make sure the destination can keep up. The rate_limit parameter of this plugin can also be used to slow down reading.

If overflow can still happen, overflow_action block is the closest thing to in_tail-style backpressure that works with this plugin today:

<buffer>
  overflow_action block
</buffer>

With block, the emit sleeps until the buffer becomes storable again, which stalls reading instead of raising, and Windows keeps the events in the channel's .evtx file, so nothing is lost as long as the channel's retention is not exceeded while blocked. Two caveats: all channel timers of one plugin instance share a single event loop thread, so blocking on one channel stalls every channel of that instance (including the refresh_subscription_interval timer), and the Fluentd docs note that block is intended for batch-like workloads rather than for throughput.

drop_oldest_chunk is the alternative if you prefer to keep the pipeline moving and can accept losing the oldest buffered data. Staying on the default throw_exception is the one option I would avoid here, unless you also set refresh_subscription_interval so that a stopped channel gets resubscribed automatically.

Follow-up

I will keep this open as an enhancement. The plugin should rescue BufferOverflowError in on_notify_*, keep the timer alive, and re-create the subscription from the persisted bookmark so that the un-emitted batch is read again, which would give behavior comparable to in_tail.