infra/monitoring/alerts.yml already fires on; change the rule and this page
is describing something else.
Availability
Target: the API is reachable 99.9% of the month (43m 43s of allowed downtime). Measured asup{job="trustowl-api"} — Prometheus’s own scrape result, not a
synthetic check. TrustOwlAPIDown fires after 2 minutes of failed scrapes,
which is the objective’s detection latency: an outage inside the error budget
is invisible for up to 2 minutes before it starts being tracked as an incident.
TrustOwlAPIErrorRateHigh). A deployment can be “up” by the reachability
objective — Prometheus can scrape /metrics — while failing this one, if the
database is unreachable and every request 500s. Both are tracked because they
fail independently: see /ready
for why a readiness failure and a scrape failure are different events.
Latency
Target: p95 API request latency under 2 seconds, over any rolling 10-minute window (TrustOwlAPILatencyHigh). This covers request-response
endpoints only — the fast path is a few milliseconds and the slow path is a
database query, never a scan, which is asynchronous and excluded from this
histogram by construction (trustowl_http_request_duration_seconds only
observes HTTP handlers; see
the metrics section).
Target: a dispatchable scan is claimed within 10 minutes of becoming
eligible (TrustOwlScanQueueAgeHigh, trustowl_scans_queued_age_seconds).
There is deliberately no target for total scan duration — pulling and
unpacking a 4 GB image is a different latency budget than a 40 MB one, and a
single number would either be so loose it catches nothing or so tight it
pages on every large image. Track trustowl_scans_duration_seconds per
target kind instead, and set a threshold against your own fleet’s image size
distribution if you want one.
What is not an SLO here
Scan failure rate (TrustOwlScanFailureRateHigh, >25% over 15 minutes) is
an anomaly threshold, not an objective — a healthy fleet’s failure rate is
driven by expired registry credentials and malformed images, which is
customer behaviour TRUSTIVAN does not control and should not be held to a
target for. It pages because a sudden rise usually means something in the
platform broke (the vulnerability database, worker egress), not because some
fraction of failures is the goal.
Backup and restore-drill freshness
(TrustOwlBackupStale, TrustOwlRestoreDrillStale) are operational
hygiene, covered in docs/deployment/backup-and-recovery.md in the repository
rather than here — they are pass/fail per run, not a rate to budget against.
Computing your own error budget
Every objective above is one Prometheus query away because the alert expressions already compute the numerator;avg_over_time or sum(rate(...))
over a longer window turns “did this fire” into “how much of the budget did it
spend”. Import infra/monitoring/dashboard.json from the repository into
Grafana, point it at the same Prometheus, and the panels already plot the raw
series; add a 30-day window panel with the formula above for the monthly
figure.