perf(ingester): invalidate head postings cache per-label instead of per-metric

The head expanded postings cache previously keyed a per-metric "seed" into
the cache key, so creating or deleting any series for a metric invalidated
every cached entry for that metric name, even entries whose result could not
have changed.

Replace the seed with a per-(tenant, label) write counter. On series
create/delete each of the series' label counts is incremented. On creation, a cache entry
snapshots the counts of its matched labels and stays valid as
long as at least one tracked count is unchanged. Creating or deleting a series which matches
all of the tracked labels expires the cache entries.

Matchers that can match an absent label (e.g. foo!="x", foo=~".*") are
excluded from the snapshot since a matching series need not carry the label,
so they cannot vouch for an entry. Name-only queries fall back to the old
whole-metric behaviour via the __name__ count.

The counter array is shared across tenants and namespaced by userId; it is
uint32 (8MB) and read under the existing stripe locks. Adds a
reason="stale" value to the cache miss metric to distinguish count
invalidation from TTL expiry.

Signed-off-by: Kyle Stang <kylestng@amazon.com>

Author: kylestang

CHANGELOG.md

@@ -36,6 +36,7 @@
 * [ENHANCEMENT] Distributor: Added `cortex_distributor_received_histogram_buckets` metric to track number of buckets in received native histogram samples before validation, per user. #7569
 * [ENHANCEMENT] Distributor: Add `WrappedHistogram` with configurable size limit (`-validation.max-native-histogram-size-bytes`) to cap native histogram protobuf size before unmarshalling. #7570
 * [ENHANCEMENT] Ingester: Add lazy regex evaluation on head postings cache miss. Defers expensive regex matchers on high-cardinality labels to per-series filtering when a selective equality matcher already narrows the result set. Configured via `-blocks-storage.expanded_postings_cache.head.lazy-matcher-max-cardinality` (disabled by default). #7553
+* [ENHANCEMENT] Ingester: Improve head expanded postings cache hit rate by invalidating only the cache entries whose matched labels changed when a series is created or deleted, instead of invalidating all entries for the metric name. #TBD
 * [ENHANCEMENT] Query Frontend: Improve the slow query log with `source`, `user_agent`, `engine_type`, `block_store_type`, and query stats fields to aid slow query diagnosis. #7601
 * [ENHANCEMENT] Ring: Add ring metric to count number of duplicate tokens. #7626
 * [ENHANCEMENT] Ring: Cache `ShuffleShardWithLookback` subrings. The cached entry is invalidated on topology change or once `now` reaches the earliest `RegisteredTimestamp + lookbackPeriod` of any included instance. #7628

pkg/ingester/ingester_test.go

@@ -6424,7 +6424,7 @@ func TestExpendedPostingsCacheIsolation(t *testing.T) {
 	cfg.BlocksStorageConfig.TSDB.BlockRanges = []time.Duration{2 * time.Hour}
 	cfg.LifecyclerConfig.JoinAfter = 0
 	cfg.BlocksStorageConfig.TSDB.PostingsCache = cortex_tsdb.TSDBPostingsCacheConfig{
-		SeedSize: 3, // lets make sure all metric names collide
+		LabelCounterSize: 3, // lets make sure all metric names collide
 		Head: cortex_tsdb.PostingsCacheConfig{
 			Enabled:  true,
 			Ttl:      time.Hour,
@@ -6925,9 +6925,9 @@ func TestExpendedPostingsCache(t *testing.T) {
 			// Query block and head
 			require.Equal(t, postingsForMatchersCalls.Load(), int64(c.expectedBlockPostingCall+c.expectedHeadPostingCall))
 
-			// Adding a metric for the first metric name so we expire all caches for that metric name
+			// Adding a series with the queried label "a" to expire the caches for that metric name and label
 			_, err = i.Push(ctx, cortexpb.ToWriteRequest(
-				[]labels.Labels{labels.FromStrings(labels.MetricName, metricNames[0], "extra", "1")}, []cortexpb.Sample{{Value: 2, TimestampMs: 4 * 60 * 60 * 1000}}, nil, nil, cortexpb.API))
+				[]labels.Labels{labels.FromStrings(labels.MetricName, metricNames[0], "a", "new", "extra", "1")}, []cortexpb.Sample{{Value: 2, TimestampMs: 4 * 60 * 60 * 1000}}, nil, nil, cortexpb.API))
 			require.NoError(t, err)
 
 			for in, name := range metricNames {