Queue Metrics (queue-wait & backlog)

Warp already tracks how long each job takes to run (execution metrics — warp.job.execution.*). Queue metrics track the other half of a job system's health: how long jobs wait before a worker picks them up, and how deep each queue is backed up.

• Queue-wait latency — the time a job spends eligible-but-unclaimed (from the moment it becomes Enqueued with ScheduleTime ≤ now to the moment a worker claims it). Recorded at the claim site, on the worker hot path, into an in-memory buffer — no extra database round-trip and no row per increment (see How counters reach the database).
• Backlog depth + oldest-age — how many eligible jobs are waiting on each queue, and the age of the oldest one. Sampled periodically off the hot path by the BacklogSampler server task.

Both are always on — no addon opt-in. The dashboard surfaces them in the Queues family on the Counters page; the meters are emitted unconditionally for any OpenTelemetry collector.

What you get

Meters (always emitted)

Meter	Type	Unit	Tags
warp.job.queue.wait	Histogram	ms	queue, application (when set)
warp.job.queue.depth	ObservableGauge	{job}	queue, application (when set)
warp.job.queue.oldest_age_seconds	ObservableGauge	s	queue, application (when set)

Meters follow the null-listener pattern — zero cost until a collector subscribes. Tags stay low-cardinality; group/PII is never a meter tag.

Durable aggregates (dashboard)

Queue-wait folds into Counter → Statistic rows the same way execution and adapter metrics do (a count, a duration-sum, and a latency histogram per queue), so avg / p95 / p99 survive raw-row cleanup. Backlog depth and oldest-age are written as gauge Statistic rows by the sampler (overwritten each tick; drained queues reset to 0).

Queue-wait on the hot path

Queue-wait is measured where a worker flips a job Enqueued → Processing. The wait is claimTime − Job.ScheduleTime (ScheduleTime advances on requeue, so a requeued job's wait is measured from its requeue, not its original enqueue). The increments are summed into a process-wide in-memory buffer — the hot path gains an interlocked add, never a query, a row or a round-trip (the worker fetch/execute path stays sacred).

They are staged during the claim and moved into the shared buffer only once the claim's transaction commits, so an attempt that rolls back contributes nothing.

How counters reach the database

Counters emitted by the worker — queue-wait, per-type and per-handler execution stats, outcome totals and their reason breakdown — take this path:

worker hot path → in-memory buffer → CounterBufferFlusher → Counter rows → CounterAggregator → Statistic rows

The buffer is the worker path only. Adapter, endpoint and client-event counters are written as Counter rows directly by their own flushers, which are already batched off the hot path and already collapse many calls into one write; saga counters, manual requeue/delete stats, crash-recovery stats and error-group trends are written directly too. Everything converges at Counter → CounterAggregator → Statistic, so the fold, the retention tiers and the dashboard are identical either way — the difference is only how increments get into the Counter table, and it matters for durability (below).

The worker hot path used to write a Counter row per increment. A finalizing job emits roughly twenty of them, and for a uniform workload they are the same twenty keys every time — so 100,000 jobs wrote two million rows that CounterAggregator then folded back down to about twenty Statistic rows. Measured on a 100k-job run, that traffic (the inserts, the aggregator's reads, and the deletes) was the single largest share of database execution time.

Increments now accumulate in memory and CounterBufferFlusher writes them out every CounterBufferFlushInterval (default 2 seconds), one row per distinct key per interval. The same 100,000 jobs wrote 1,799 rows instead of two million. Nothing downstream changed: CounterAggregator sees rows of identical shape, there are simply far fewer of them carrying larger values.

Counters are not readable immediately after the job that emitted them

A worker used to write its Counter rows inside the same transaction that finalized the job, so a reader saw the increment the moment the job reached Completed. It does not any more — the increment is in memory until the next flush:

// job completes, then immediately:
var stat = await db.Set<Statistic>().FirstAsync(x => x.Key == "stats:succeeded");
// may not include that job, for up to CounterBufferFlushInterval

There is no error; the read just returns the earlier value. These are cumulative totals, so anything that polls or retries converges on its own — which covers dashboards, since the fold into Statistic already lags a full CounterAggregationInterval anyway. If you genuinely need read-after-write, set CounterBufferFlushInterval low (say 200 ms) and give back a proportional share of the write reduction.

The durability trade

Worker counter increments live in memory until the next flush. An ungraceful exit — kill -9, an OOM, a lost container — loses at most one interval of them. A graceful shutdown flushes before exiting, so an orderly stop loses nothing.

This applies to the worker-emitted counters listed above and to nothing else. Adapter, endpoint and client metrics keep whatever durability their own lossy channels already gave them, which this change did not touch.

This is deliberate, and it is the reason Counter exists as a separate table from Statistic in the first place: counters are the write-optimised, lossy side of the metrics fold. If a number must survive a crash, it belongs in a row, not a counter. Job state, logs, and every control-plane record are unaffected — they were never counters.

Shorten CounterBufferFlushInterval to narrow the loss window; lengthen it to write fewer, larger rows. A non-positive value is rejected at AddWarpServer (it is the only delay in the flusher's loop, so zero would spin it).

Aggregator drains are bounded

CounterAggregator takes at most 20 batches of 1,000 rows per tick and asks to be re-run immediately if more remain. It used to drain in a single while (true) inside the task host's lock transaction, which on a large backlog held one transaction open for as long as the drain took — measured at 51 minutes against a 500,000-job backlog.

That matters far beyond the aggregator: an open transaction pins the vacuum horizon, so autovacuum cannot reclaim any dead tuple newer than it. The job table grew, every worker claim scanned more dead rows, and throughput fell from about 500 jobs/sec to 27 while the queue was still draining — the aggregator running longer made the bloat worse, which made the aggregator slower. The same work now happens at the same rate in short transactions, so the horizon keeps advancing.

Backlog sampling (off the hot path)

BacklogSampler is an ordinary IServerTask (like CounterAggregator) that runs every BacklogSampleInterval (default 60s), takes a distributed lock, and runs one grouped query:

SELECT queue, COUNT(*), MIN(schedule_time)
FROM job
WHERE kind = Job AND current_state = Enqueued AND schedule_time <= now
GROUP BY queue

This is served by the existing (Kind, CurrentState, Queue, ScheduleTime) index — no new index, no hot-path cost. The result updates the always-on gauges and the qbacklog: Statistic rows.

Scheduled jobs and the (CurrentState, ScheduleTime) index

Queue-wait is measured from a job's ScheduleTime, so for any job that was scheduled rather than enqueued immediately — a retry backoff, a webhook retry, a saga timeout, a rate-limit Wait reschedule — the activation task's latency is part of the queue-wait you see on this page. That makes the scheduling path worth understanding here.

ScheduledJobActivation runs one statement every ScheduledActivationInterval (default 10s):

UPDATE job SET current_state = 1
WHERE current_state = 7 AND schedule_time <= now()
RETURNING id, queue, schedule_time

Why it does not filter on Kind

The backlog query above filters kind = Job and so rides the existing (Kind, CurrentState, Queue, ScheduleTime) index. The activation statement cannot, because it deliberately does not constrain Kind.

Publisher parks a delayed ITimeoutMessage as Kind = Message in State.Scheduled — that is how saga timeouts fire. Narrowing the activation predicate to Kind = Job so it could reuse the existing index would strand every saga timeout forever, with no error and no log line: the rows would sit in Scheduled and never activate. A Kind IN (Job, Message) variant would work today but re-breaks the moment any future path parks a third kind in Scheduled, and it measured roughly 10× slower than the dedicated index anyway.

So the predicate stays broad — CurrentState and ScheduleTime only — and Warp carries an index shaped to it:

job.HasIndex(p => new { p.CurrentState, p.ScheduleTime });

Requires a migration

This index is picked up by a standard dotnet ef migrations add / database update. On a large job table, note that a plain CREATE INDEX blocks writes for the duration of the build — see the release notes for the CONCURRENTLY / ONLINE = ON workaround.

The SQL Server provider's activation statement carries the identical predicate, so the same index serves both providers.

What it costs and what it buys

Without a CurrentState-leading index the planner falls back to (Kind, CurrentState, CreateTime) on current_state alone, then heap-fetches every row in Scheduled and discards the future-dated ones. Measured on 250,000 job rows with 5,675 in Scheduled: 31.8 ms and 2,686 buffers per tick, versus 0.35 ms and 4 buffers with the index, purely from eliminating 2,841 rows fetched-and-discarded.

The write side pays for it: +17.8% on a bulk insert and +8.9% on a bulk state transition, which over a job's whole life (one insert plus ~3 transitions) is ≈ 5 µs of database time per job. Index size is 9.2 MB against 71 MB of existing job indexes. The worker's own claim query was re-checked with and without the index — identical plan, no regression.

When it is not worth it

There is no jobs/sec break-even. Write cost scales with your job rate; read cost scales with how many rows are sitting in Scheduled, which is itself your job rate times the fraction delayed times the average delay. Throughput appears on both sides and cancels.

The deciding variable is how long work sits in Scheduled — not how fast it moves. Warp's own defaults keep that number high: retry delays are [15, 60, 300] seconds, the webhook retry schedule is [1m, 10m, 1h, 6h] (a delivery in its six-hour backoff holds a Scheduled row for six hours), and saga timeouts and rate-limit Wait reschedules park rows there too.

If you run no retries, no webhooks, no sagas and no scheduled jobs, nothing accumulates in Scheduled, the read cost the index removes was never being paid, and you are buying ~18% insert overhead for nothing. That is a legitimate reason to drop the index from your migration.

Full EXPLAIN plans, buffer counts and the A/B write measurements are in docs/perf-results.md in the repository.

Sinks

Queue-wait respects WarpConfiguration.JobMetricsSink (observability sinks) exactly like execution metrics:

• Database (default) / Both — Counter rows are written; the dashboard's Queues counter family works.
• Otel — the counter increments are skipped entirely (no buffer entry, no row); the warp.job.queue.wait meter still fires, carrying the data to your collector. Use Grafana/Prometheus for percentiles; the dashboard page will have no rows.

The backlog gauges always emit; the qbacklog: Statistic upsert is likewise skipped under Otel.

Multi-application

When ApplicationName is set, queue-wait is additionally sliced by the executor application under a disjoint counter-key namespace (qwait-app:), and application becomes a meter tag. The per-application slice carries no latency histogram (to bound counter volume), so its percentiles read as 0 — averages and counts are still exact. Pass ?application=<name> to the API to read a single app's queue-wait.

Backlog is not sliced by application — it's a queue-global signal. An eligible-but-unclaimed job has no executor yet, and slicing by the publishing app would be a per-creator metric, which Warp deliberately omits (§8.23). So the ?application= view reports that app's queue-wait alongside the queue's overall backlog depth and oldest-age, and the backlog gauges carry only a queue tag.

API

• GET {prefix}/api/queues/metrics — retained as public read API (the dashboard reads the counter aggregates directly). Per-queue rows: claimedCount, avgWaitMs, p95WaitMs, p99WaitMs, backlogDepth, oldestAgeSeconds. Optional ?application= filter. Resolves in any AddWarp process (dashboard-only / publisher-only included) since it reads only the durable aggregates.

Outcome stats: a metric records an event, current state is a query

The stats: counter family answers what has happened, and it is append-only — nothing rewrites it. A job that failed, was requeued, and then succeeded is counted in both stats:failed and stats:succeeded, because both events happened.

What is happening right now is a different question with a different source: the dashboard tiles and navigation badges count rows in the Job table. The two are not redundant. Completed jobs are swept at JobExpirationTimeout, so a query can never answer "how many ever succeeded"; and a cumulative counter can never answer "how many are failed at this moment".

This is why a requeue decrements nothing. Earlier versions decremented the source state's counter, which made a lifetime total disagree with the sum of its own hourly buckets (the decrement wrote no bucket row), let the dashboard throughput chart see a negative delta, and left the counter answering roughly the same question as the live query while losing the only thing it uniquely provides.

The three levels

Key	Level	Meaning
stats:unsuccessful	umbrella	Every terminal outcome that is not Completed. Derived, not stored — see below
stats:succeeded, stats:failed, stats:deleted, stats:requeued	state total	One row per outcome event
stats:{state}-{reason}	breakdown	Why it happened
stats:retried-jobs	standalone	Distinct jobs that entered retry, not retry events

stats:retried-jobs sits outside the hierarchy on purpose — it counts jobs where every other key counts events. A job retried fifteen times contributes fifteen to stats:requeued-retry and one to stats:retried-jobs, so the pair answers "how much retrying is happening" and "how many jobs are thrashing" separately. It increments only on a job's first retry, detected from the retry attempt count the worker already reads before running the handler — so it costs no extra read and no schema column.

Each state total is written independently of its breakdown, so a reader never sums the parts to get a total. The breakdown usually totals less than its state key: an outcome with no attributable cause carries no reason, and that difference is the unattributed remainder — surfaced rather than hidden, since a breakdown that silently fails to add up reads like a bug.

The umbrella is computed on read as failed + deleted, and no stats:unsuccessful row is ever written. Ten different sites move those two keys — both worker paths, DeleteJob, BulkDeleteJobs, crash recovery's cancel and fail arms, worker cancellation — and a stored umbrella has to be maintained at every one of them or it under-reports. Deriving it puts the definition in one place, where it cannot drift from the totals it sums.

Only failed and deleted sit under the umbrella. A requeue is not a terminal outcome — the same job runs again and lands in one of the other totals — so it is a top-level key, not an unsuccessful one.

Reasons

OutcomeReason is a closed enum, stamped on JobOutcome by the pipeline behaviour that made the decision — the only component that knows why. Its wire token is pinned by a test, so renaming an enum member cannot silently rename a live metric key and orphan its history.

Reason	Token	Written when
Retry	retry	Retry backoff scheduled another attempt
RetryExhausted	retry-exhausted	The retry budget ran out; this failure is terminal
Concurrency	concurrency	Mutex / semaphore — Wait requeues, Skip deletes
RateLimit	ratelimit	Throttled, skipped, or bounced off lock contention
Timeout	timeout	Timeout in Delete mode
Saga	saga	Busy, version conflict, missing correlation, or a moot timeout
Manual	manual	Operator requeue from the dashboard
Recovery	recovery	Crash recovery re-queued work whose worker stopped responding

A custom pipeline behaviour may set Reason on any outcome it constructs:

_jobContext.Outcome = new JobOutcome
{
    State = State.Deleted,
    Reason = OutcomeReason.RateLimit,
    LogMessage = "Dropped — vendor quota exhausted",
};

It is an enum rather than a string on purpose. A free-form reason would let one caller mint a Statistic row per tenant, per key, or per URL; the bounded set is what keeps this family a fixed number of keys regardless of traffic.

Manual and Recovery are the two members not stamped on a JobOutcome — no pipeline runs for them. The dashboard requeue paths and crash recovery write their keys directly, using the same tokens so the breakdown reads uniformly.

An unmapped reason degrades, it does not throw

The token lookup returns the literal unknown for a reason it has no mapping for, rather than throwing. This is not defensive habit — a throw there is genuinely unsafe:

Every caller is a finalization site running inside the job's own catch. Throwing from it would be laundered into a fake handler failure, re-enter finalization from that catch, throw again, and escape before the state is saved. The job would stay Processing, crash recovery would requeue it, and it would re-poison itself indefinitely — a metrics-lookup miss would have taken down job processing. Instrumentation never out-throws in Warp, the same rule adapters and the notifier seam follow.

The fallback is a fixed literal rather than anything derived from the unmapped value, so the miss costs one extra bounded key instead of a new key family. A guard test asserts every enum member maps to a distinct token other than unknown, so a member added without a token fails the test run — the literal exists for runtime safety, not as a substitute for the mapping.

Meter

warp.job.requeued is a Counter<long> emitted unconditionally for every requeue — worker finalization (retry, mutex Wait, rate-limit Wait, circuit breaker), dashboard requeues (reason=manual), and crash recovery (reason=recovery) — tagged job.type, queue, reason, and application (when set). The manual and recovery paths emit after their transaction commits, so a rollback can never leave the meter above the stats:requeued counter it mirrors.

It closes a real gap: concurrency and rate limiting already emitted detailed spans, but spans are sampled, so "how many jobs bounced off this mutex in the last hour" was not answerable from telemetry at all. The concurrency and rate-limit keys are deliberately not tags — they are unbounded and PII-adjacent, and stay on the span where cardinality does not compound.

Mutex and semaphore share one reason

Both go through the same behaviour and take the same lock, and the only local discriminator is whether the effective limit is 1 — which an admin limit override can invert. A label derived from it would be wrong exactly when someone is tuning limits, so both report concurrency. The distinction you usually want falls out of the outcome anyway: Wait mode requeues, Skip mode deletes.

Cost

Flat keys do not scale with throughput — a million jobs a day and a hundred produce the same number of rows, only different values. Each key retains one lifetime row, 168 hourly buckets (7 days), and 83 daily buckets, so the whole family is a few thousand rows regardless of load. A completed job writes exactly what it wrote before this feature existed; only outcomes carrying a reason write the extra breakdown row.