Skip to content

perf: experiment with striped classic histogram accumulator#2329

Draft
zeitlinger wants to merge 2 commits into
prometheus:mainfrom
zeitlinger:agent/classic-only-striped-accumulator
Draft

perf: experiment with striped classic histogram accumulator#2329
zeitlinger wants to merge 2 commits into
prometheus:mainfrom
zeitlinger:agent/classic-only-striped-accumulator

Conversation

@zeitlinger

Copy link
Copy Markdown
Member

Summary

This is an experimental alternate implementation for benchmarking against main and the synchronized candidate in #2252. It is not being presented as production-ready yet.

For classic-only histograms, this replaces the existing Buffer/LongAdder recording path with an epoch-striped, double-buffered per-thread accumulator. Each recording thread owns a cell, observations update plain thread-confined bucket/count/sum fields, and collection advances a global epoch before draining the inactive buffers. This avoids a global synchronized (DataPoint) monitor on the shared four-thread observe path while preserving coherent bucket/count/sum snapshots. getCount() and getSum() use the same snapshot mechanism.

Native-only and hybrid histograms keep their existing implementation. The public API is unchanged.

Local evidence

  • mise run lint
  • ./mvnw -pl prometheus-metrics-core -Dtest=HistogramTest test — 35 tests passed
  • focused concurrent collect/lifecycle tests repeated 5 times
  • ./mvnw -pl prometheus-metrics-core -am verify — reactor passed; core ran 157 tests
  • git diff --check
  • representative local JMH smoke run (JDK 25.0.3, 4 threads, 1 fork, 3x1s warmup, 5x1s measurement):
    • HistogramBenchmark.prometheusClassic: 40,944.515 ± 302.666 ops/s

The local JMH number is a smoke result, not a controlled cross-branch conclusion. The benchmark workflow is being requested so this branch and main are measured on the same runner.

Added concurrency coverage

  • collection while eight shared observers record concurrently
  • monotonic and coherent bucket/count/sum snapshots
  • exact final getCount() / getSum() values
  • retention of observations from short-lived recording threads
  • clear() lifecycle behavior for old and newly-created data points

Remaining hardening before merge consideration

  • Review/validate the epoch handoff under the Java Memory Model, ideally with dedicated jcstress coverage rather than relying only on repeated JUnit stress tests.
  • Decide how to bound or reclaim retained per-thread cells for workloads with high thread churn; cells currently live for the data point lifetime so completed-thread observations cannot be lost.
  • Measure scrape latency and the spin-wait behavior when a recording thread is delayed after entering an epoch.
  • Measure the cost of getCount() / getSum(), which currently perform an epoch flip and cell scan to return exact accumulated values.
  • Run the full CI benchmark matrix and test additional supported JDKs/architectures before treating this as mergeable.

Related: #2252

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ API changes detected — maintainer review required

This PR modifies the published API diff for the following module(s):

  • prometheus-metrics-core

Please review the changes in docs/apidiffs/current_vs_latest/ carefully before approving.

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ API changes detected — maintainer review required

This PR modifies the published API diff for the following module(s):

  • prometheus-metrics-core

Please review the changes in docs/apidiffs/current_vs_latest/ carefully before approving.

@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Benchmark results

Benchmark run succeeded for 6fe299560f57865c3ca33d399163a0639f84b47e.

Prometheus Java Client Benchmarks

Run Information

  • Date: 2026-07-23T10:46:17Z
  • Commit: 6fe2995
  • JDK: 25.0.3 (OpenJDK 64-Bit Server VM)
  • Benchmark config: 3 fork(s), 3 warmup, 5 measurement, 4 threads
  • Hardware: AMD EPYC 9V74 80-Core Processor, 4 cores, 16 GB RAM
  • OS: Linux 6.17.0-1020-azure

Comparison with base

  • Head: 6fe2995
  • Base: 1ca2716
  • Change: positive means the PR is faster than base.
  • Note: Base and head are compared on the same runner within each topic.
Benchmark PR Base Change Result
CounterBenchmark.codahaleIncNoLabels 44.11K 44.37K -0.6% within noise
CounterBenchmark.openTelemetryAdd 3.25K 4.53K -28.2% slower
CounterBenchmark.openTelemetryInc 4.42K 4.49K -1.6% within noise
CounterBenchmark.openTelemetryIncNoLabels 3.59K 4.70K -23.7% within noise
CounterBenchmark.prometheusAdd 48.73K 48.49K +0.5% within noise
CounterBenchmark.prometheusInc 60.03K 59.25K +1.3% within noise
CounterBenchmark.prometheusNoLabelsInc 51.95K 51.05K +1.8% faster
CounterBenchmark.simpleclientAdd 6.12K 5.92K +3.4% faster
CounterBenchmark.simpleclientInc 6.16K 6.20K -0.5% within noise
CounterBenchmark.simpleclientNoLabelsInc 5.92K 5.92K +0.1% within noise
HistogramBenchmark.openTelemetryClassic 766.00 766.29 -0.0% within noise
HistogramBenchmark.openTelemetryExponential 628.37 704.07 -10.8% within noise
HistogramBenchmark.prometheusClassic 11.59K 7.18K +61.4% faster
HistogramBenchmark.prometheusClassicPerThread 12.23K 12.58K -2.8% slower
HistogramBenchmark.prometheusClassicSingleThread 4.40K 4.55K -3.2% slower
HistogramBenchmark.prometheusNative 2.81K 2.82K -0.3% within noise
HistogramBenchmark.simpleclient 4.41K 4.36K +1.2% within noise
HistogramTextFormatBenchmark.openMetricsWriteToNull 27.07K 27.24K -0.6% within noise
HistogramTextFormatBenchmark.prometheusWriteToNull 27.56K 27.63K -0.2% within noise
TextFormatUtilBenchmark.openMetricsWriteToByteArray 526.71K 533.80K -1.3% within noise
TextFormatUtilBenchmark.openMetricsWriteToNull 541.08K 546.14K -0.9% within noise
TextFormatUtilBenchmark.prometheusWriteToByteArray 568.30K 573.63K -0.9% within noise
TextFormatUtilBenchmark.prometheusWriteToNull 582.87K 570.66K +2.1% within noise

Results for PR head

CounterBenchmark

Benchmark Score Error Units Within run
prometheusInc 60.03K ± 926.53 ops/s fastest
prometheusNoLabelsInc 51.95K ± 18.60 ops/s 1.2x slower
prometheusAdd 48.73K ± 396.08 ops/s 1.2x slower
codahaleIncNoLabels 44.11K ± 115.85 ops/s 1.4x slower
simpleclientInc 6.16K ± 130.99 ops/s 9.7x slower
simpleclientAdd 6.12K ± 22.12 ops/s 9.8x slower
simpleclientNoLabelsInc 5.92K ± 18.44 ops/s 10x slower
openTelemetryInc 4.42K ± 1.19K ops/s 14x slower
openTelemetryIncNoLabels 3.59K ± 261.88 ops/s 17x slower
openTelemetryAdd 3.25K ± 151.58 ops/s 18x slower

HistogramBenchmark

Benchmark Score Error Units Within run
prometheusClassicPerThread 12.23K ± 292.11 ops/s fastest
prometheusClassic 11.59K ± 428.04 ops/s 1.1x slower
simpleclient 4.41K ± 76.08 ops/s 2.8x slower
prometheusClassicSingleThread 4.40K ± 19.23 ops/s 2.8x slower
prometheusNative 2.81K ± 252.77 ops/s 4.4x slower
openTelemetryClassic 766.00 ± 23.21 ops/s 16x slower
openTelemetryExponential 628.37 ± 82.07 ops/s 19x slower

HistogramTextFormatBenchmark

Benchmark Score Error Units Within run
prometheusWriteToNull 27.56K ± 317.22 ops/s fastest
openMetricsWriteToNull 27.07K ± 332.83 ops/s 1.0x slower

TextFormatUtilBenchmark

Benchmark Score Error Units Within run
prometheusWriteToNull 582.87K ± 12.95K ops/s fastest
prometheusWriteToByteArray 568.30K ± 5.31K ops/s 1.0x slower
openMetricsWriteToNull 541.08K ± 12.50K ops/s 1.1x slower
openMetricsWriteToByteArray 526.71K ± 15.29K ops/s 1.1x slower

Raw Results

Benchmark                                            Mode  Cnt          Score        Error  Units
CounterBenchmark.codahaleIncNoLabels                thrpt   15      44112.334    ± 115.847  ops/s
CounterBenchmark.openTelemetryAdd                   thrpt   15       3252.670    ± 151.579  ops/s
CounterBenchmark.openTelemetryInc                   thrpt   15       4416.392   ± 1185.330  ops/s
CounterBenchmark.openTelemetryIncNoLabels           thrpt   15       3586.093    ± 261.881  ops/s
CounterBenchmark.prometheusAdd                      thrpt   15      48729.646    ± 396.084  ops/s
CounterBenchmark.prometheusInc                      thrpt   15      60027.803    ± 926.533  ops/s
CounterBenchmark.prometheusNoLabelsInc              thrpt   15      51952.103     ± 18.603  ops/s
CounterBenchmark.simpleclientAdd                    thrpt   15       6118.802     ± 22.118  ops/s
CounterBenchmark.simpleclientInc                    thrpt   15       6163.636    ± 130.990  ops/s
CounterBenchmark.simpleclientNoLabelsInc            thrpt   15       5922.243     ± 18.435  ops/s
HistogramBenchmark.openTelemetryClassic             thrpt   15        765.996     ± 23.210  ops/s
HistogramBenchmark.openTelemetryExponential         thrpt   15        628.374     ± 82.069  ops/s
HistogramBenchmark.prometheusClassic                thrpt   15      11589.781    ± 428.043  ops/s
HistogramBenchmark.prometheusClassicPerThread       thrpt   15      12228.590    ± 292.114  ops/s
HistogramBenchmark.prometheusClassicSingleThread    thrpt   15       4398.687     ± 19.226  ops/s
HistogramBenchmark.prometheusNative                 thrpt   15       2808.686    ± 252.769  ops/s
HistogramBenchmark.simpleclient                     thrpt   15       4410.533     ± 76.084  ops/s
HistogramTextFormatBenchmark.openMetricsWriteToNull  thrpt   15      27071.116    ± 332.829  ops/s
HistogramTextFormatBenchmark.prometheusWriteToNull  thrpt   15      27561.877    ± 317.215  ops/s
TextFormatUtilBenchmark.openMetricsWriteToByteArray  thrpt   15     526713.833  ± 15292.911  ops/s
TextFormatUtilBenchmark.openMetricsWriteToNull      thrpt   15     541078.028  ± 12499.428  ops/s
TextFormatUtilBenchmark.prometheusWriteToByteArray  thrpt   15     568297.769   ± 5312.819  ops/s
TextFormatUtilBenchmark.prometheusWriteToNull       thrpt   15     582874.507  ± 12954.847  ops/s

Notes

  • Score = Throughput in operations per second (higher is better)
  • Error = 99.9% confidence interval
  • Comparison with base uses JMH confidence intervals when available; overlapping intervals are marked "within noise".
  • Within run compares benchmarks in the same result set, not against the base commit.

Benchmark Descriptions

Benchmark Description
CounterBenchmark Counter increment performance: Prometheus, OpenTelemetry, simpleclient, Codahale
HistogramBenchmark Histogram observation performance (classic vs native/exponential)
TextFormatUtilBenchmark Metric exposition format writing speed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant