Skip to main content

Metrics retention tiers

Warp folds every time-series metric — per-type/handler execution (jobstat), queue-wait (qwait), adapter and endpoint call stats, browser events, error-group trends — through one pipeline: hot-path code emits counter increments, they land in the Counter table (worker metrics via an in-memory buffer flushed on an interval, everything else written as rows by its own flusher), a CounterAggregator sums them into durable Statistic rows, and the dashboard reads those back. Metrics retention tiers control how that time-series data ages: recent data is kept at fine (5-minute) resolution, and a background task progressively rolls it up into hourly then daily buckets as it ages — so recent graphs stay detailed while long history stays cheap.

The model​

Every bucketed key carries an explicit resolution marker and a stamp:

fine   (5-min): {family}:…:hist:{token}:m5:2026-08-02-14-25
hourly (1-hr):  {family}:…:hist:{token}:h1:2026-08-02-14
daily  (1-day): {family}:…:hist:{token}:d1:2026-08-02

Latency percentiles get the same treatment under a pcth marker, so a windowed p95/p99 can be computed over any tier. Lifetime totals (…:{token} with no date) and the lifetime pct histogram are untouched — they accumulate forever and back the all-time dashboard numbers.

The write path emits at the fine tier (same number of counter rows as before — just a finer stamp), and the StatisticRollup server task does the rest:

  1. fine → hourly — 5-minute buckets older than FineResolutionRetention (default 6h) are summed into their hour and deleted.
  2. hourly → daily — hourly buckets older than HourlyStatisticsRetention (default 7d) are summed into their day and deleted.
  3. prune — daily buckets older than DailyStatisticsRetention (default 90d) are deleted.

The roll sums into the coarser bucket before deleting the finer one, inside the task's lock transaction — so a crash mid-run can never double-count, and no detail is dropped (unlike the old prune, which simply deleted expired hourly rows). It runs on StatisticRollupInterval (default 10 minutes), off the worker hot path.

One tick rolls at most 20,000 source rows, shared across both passes, and re-runs back-to-back while a backlog remains. The cap is not a throttle you should want removed: the whole of the task runs inside one lock transaction, and an open transaction pins the vacuum horizon for as long as it lasts — the failure mode is a rollup that holds a transaction open long enough for dead tuples to accumulate in the job table, which slows the very workers the metrics describe. Deferring is safe because nothing is dropped: a fine bucket held back by the cap whose hourly parent was rolled away in the same tick simply re-creates that hourly bucket next tick and rolls on the one after, preserving the summed value. The tick you are most likely to notice is the first after upgrading to 3.10 or later, when every legacy hourly row rolls to daily at once — it now spreads across consecutive ticks instead of running as one long transaction.

Why it exists​

  • Bounded storage — instead of hourly buckets accumulating (or being deleted outright), each dimension keeps a fixed number of buckets: a few hours of 5-minute, a few days of hourly, then daily. Old history is retained coarsely rather than lost.
  • Sub-hour resolution — recent data at 5-minute granularity is what short-window queries (and SLO fast-burn alerting) need; nothing finer than an hour existed before.
  • One implementation for every family — the rollup keys purely on the trailing tier suffix, so it downsamples jobstat, qwait, adapter, endpoint, client-event, and error-group trends alike, and migrates every pre-existing hourly row to the new scheme automatically.

Configuration​

services.AddWarp<AppDb>(opt =>
{
    opt.UsePostgreSql();

    opt.FineResolutionMinutes = 5;                              // fine bucket width; 60 emits hourly directly
    opt.FineResolutionRetention = TimeSpan.FromHours(6);        // fine → hourly age
    opt.HourlyStatisticsRetention = TimeSpan.FromDays(7);       // hourly → daily age
    opt.DailyStatisticsRetention = TimeSpan.FromDays(90);       // daily prune age; null keeps daily forever
    opt.StatisticRollupInterval = TimeSpan.FromMinutes(10);     // how often the rollup runs
});

FineResolutionMinutes must be >= 1 (it's a divisor when bucketing; the server fails fast at startup otherwise), and the retentions must widen — FineResolutionRetention < HourlyStatisticsRetention < DailyStatisticsRetention. Larger FineResolutionMinutes coarsens the fine tier but doesn't turn it off — the tier is always emitted (marked m5) and downsampled. Leaving DailyStatisticsRetention = null keeps the coarse daily history indefinitely. StatisticRollup is the only pruner of time-bucketed rows, so disabling it (StatisticRollupInterval = null) lets those buckets accumulate unbounded unless you prune the Statistic table by other means.

Migration​

No schema change and no migration — every tier lives in the existing Statistic / Counter tables as new key-prefixed rows. Pre-3.10 keys (bare …:yyyy-MM-dd-HH) are recognized as hourly and migrated to daily by the rollup on their normal schedule, so upgrading is transparent. The dashboard's counters and stats-history graphs read across the tiers unchanged.