Liveness and readiness are not the same question
They are separate endpoints because they answer separate questions, and conflating them causes an outage rather than preventing one./health checks nothing, deliberately
/ready checks the dependencies
/health, readiness on
/ready. Pointing liveness at /ready reintroduces exactly the failure mode
/health exists to avoid.
/api/v1/system/status
The authenticated view, and the one that tells a user something the probes
cannot: whether the advisory data behind their results is trustworthy.
Everything it reports is the caller’s own tenant’s — queue depth, scan
counts, overdue schedules. A global queue depth would be more useful to an
operator and would also tell every customer how much work every other customer
is doing, which is a business disclosure wearing an operations badge. Global
figures belong in metrics, behind an operator’s own authentication.
A tenant’s status is reported unhealthy when the advisory data is not
trustworthy. Reporting the process healthy while it can only produce results it
has to caveat would make the endpoint agree with the failure it exists to catch.
See Vulnerability database.
The scanning engine is never named in the payload. A CI guard fails the build if
its identity reaches the API layer, and a test asserts the same thing about the
response body.
Metrics
Why it is a separate, private listener
Metrics are cross-tenant aggregates: how many scans run, how many findings exist, how often authentication fails. Every one of those is a fact about the business rather than about a tenant, so none of them belongs on a customer-facing API even behind an authorisation check. TRUSTIVAN has no operator plane, and inventing one purely to guard a route would be a worse answer than not exposing the route publicly. So the metrics live on their own listener, bound to loopback. Binding this to0.0.0.0 publishes those aggregates to anything that can route
to the host. If you need remote scraping, route to it deliberately — through a
sidecar, an overlay network, or an authenticating proxy — rather than by
widening the bind address.
Gauges are refreshed on a timer, not computed on scrape
Values are refreshed every 30 seconds by a periodic task rather than queried during a scrape. A collector that queries the database while Prometheus is scraping turns an unreachable database into a hung scrape, so monitoring goes blind at exactly the moment it is needed — and it lets anyone who can reach the port generate database load. The cost is that a gauge can be up to 30 seconds behind, which is the right trade for a number you alert on over minutes.Label cardinality is a hard constraint
A Prometheus label with unbounded values is not a slow query, it is an out-of-memory: every series is retained. Route labels therefore use the registered pattern (/api/v1/findings/:id), which is the difference between a dozen series and one
per finding ever viewed. Unmatched requests collapse to a single label, because
a 404 sweep is exactly the traffic that would otherwise mint a series per URL an
attacker tries. A test asserts that no metric carries a tenant, user, finding,
asset or image label.
If you add a metric, that constraint applies to you too.
The metric worth alerting on
trustowl_scheduler_lag_seconds.
It is the one that says the scheduler has stopped, which a queue depth of
zero cannot: a dead scheduler produces no work, so every other figure then looks
healthy. An empty queue and a broken scheduler are otherwise indistinguishable.
Alongside it, worth watching:
- Queue depth trending up while worker count is flat — add worker replicas, see Scaling.
- Scan failure rate, which distinguishes a registry problem from a capacity problem.
- Anything indicating the vulnerability database is ageing past
VULN_DB_MAX_AGE_HOURS, which means blocked egress, a stale mirror, or a cache directory that is not actually persisted.
Tracing
Spans are created throughout both the request and the scan pipeline —
HTTP handler, service, repository, scanner engine, job worker — regardless of
this setting: leaving it unset means nothing is ever exported, not that
nothing is instrumented. There is no separate
TRACING_ENABLED flag; the
endpoint IS the switch, so there is exactly one way to misconfigure this (a
typo’d host) rather than two.
Every span carries tenant.id once a request or scan is attributed to one,
and a scan’s spans additionally carry scan.id. Never a secret, customer
code, or scanned content — the same rule the metrics labels above follow, for
the same reason: a trace is retained and read by whoever operates the
collector, which for a self-hosted deployment is a third party more often
than not.
Logs
Both processes log structured events. Three lines worth recognising:
Failure messages stored on a scan are sanitised before they are written, because
they are returned by the API: registry client errors routinely embed the URL
they were fetching, and a registry URL can carry a bearer token in a query
parameter. Anything matching a marker is replaced wholesale rather than
trimmed — trimming invites a near-miss that leaks the tail of a token — control
characters are stripped so an error cannot forge extra log lines, and the
message is capped at 300 characters.
What is not here yet
Alerting and the Grafana dashboard ship in
infra/monitoring/ in the
repository (alerts.yml, dashboard.json) — see
Service level objectives for the thresholds they encode.
Next
- Upgrades — what to watch during one.
- Troubleshooting scans — when a symptom is a scan rather than the platform.