feat(plugins/enrichment): CSV lookup enrichment plugin#2365
Open
JocLRojas wants to merge 3 commits into
Open
Conversation
New Go plugin com.utmstack.enrichment that enriches log events by performing exact-match lookups against CSV files dropped in the filesystem at /workdir/pipeline/csv-datasets/. Invoked from filter YAMLs via the dynamic step. Filesystem is the source of truth: files are picked up automatically within 30s of being dropped, modified or deleted (polling watcher, no fsnotify to stay compatible with shared filesystems in multi-node deploys). Params for the dynamic step: dataset, source, match_column, output_column, destination. All required. Match values are normalised (lowercase + whitespace collapsed to underscore) both when indexing the CSV and when reading the source event field, so CSV maintainers and event producers can be inconsistent about case and spacing. Guardrails: 10000 rows hard limit per CSV (skip with ERROR), 5000 row warn threshold, UTF-8 required. Invalid or oversized files are skipped without affecting other datasets. Fail-open on any lookup miss (dataset not registered, column not present, no row match) so events keep flowing. Log dedup ensures warnings fire once per dataset id and errors once per (dataset, column) tuple to avoid log flood. Concurrency: registry uses atomic-swap pattern (readers hold an immutable *Dataset pointer). Per-column indices built lazily on first lookup and cached under a per-dataset RWMutex.
Copies the com.utmstack.enrichment.plugin binary into /workdir/plugins/utmstack/ so the event-processor supervisor picks it up on container start and the parsing plugin exposes its Unix socket for the dynamic step.
Adds the go build step for plugins/enrichment so the binary produced during the Build Plugins job is available for the event-processor Docker image COPY.
🛑 AI review — Engineer review requiredThis PR touches critical paths or introduces changes the model cannot judge with sufficient confidence. @Kbayero @osmontero please review. ✅
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
New plugin
com.utmstack.enrichmentthat enriches events by looking up values against CSV files dropped in the event-processor filesystem (/workdir/pipeline/csv-datasets/).It is invoked from filter YAMLs via the
dynamicstep with these params:dataset,source,match_column,output_column,destination.Example usage in a filter to enrich events:
Why
Currently, enriching events using CSV dictionaries is not supported. This configuration requires an external tool. This plugin integrates this functionality directly into the event processor, eliminating reliance on external services.
How it works
Guarantees
Issue
No related issue.