An upgrade replaces the image and restarts the processes. Migrations run automatically as each one starts, so there is no separate schema step — which makes the backup step the one you must not skip.

Before you start

Back up PostgreSQL. It holds every scan, finding, triage decision, accepted risk, policy verdict and schedule in the deployment, and a migration applied to it is applied for real.
Every migration has a down, so the schema is reversible in principle. But a down returns the schema, not the data a column drop took with it. The backup is what makes a rollback recover the second thing as well as the first. You do not need to back up Redis or the scanner cache: Discarding scanner_cache costs one slow scan while roughly 50 MB is re-fetched. That is a reasonable thing to do deliberately if you suspect a corrupt cache; it is not something to do routinely, because a worker that starts with a cold cache also starts with no vulnerability database, and the first scan pays for both.

The upgrade

Each process applies pending migrations before it serves anything. The runner takes a PostgreSQL advisory lock around the whole run, so several instances restarting at once serialise rather than racing, and a schema that is already current is a no-op. If a migration fails, the process exits with running migrations: … and does not serve traffic. That is the intended behaviour: a binary that finds a schema it does not understand should refuse to start rather than produce confusing runtime errors later.

Order of restart

Restart workers before or after the API — either is safe, and neither needs coordination.
  • A worker killed mid-scan has its lease expire, and the reaper returns the scan to queued. An abrupt stop costs one lease interval, not one lost scan.
  • A worker that shuts down on a signal leaves the interrupted run claimed and lets its lease expire, so the reaper returns it to queued intact — the same treatment a killed worker gets. It is deliberately not recorded as a failure: doing so would spend one of the scan’s attempts on work that was never actually tried, and three rolling restarts hitting the same scan would exhaust its retry budget and fail it permanently.
  • An API restart loses nothing: the scan row is committed before the response is returned, and the dispatcher republishes anything whose wakeup was lost.
If you run a single-process deployment (SCAN_WORKER_ENABLED=true on the API), the same applies — the restart is just one process instead of two.

What downtime costs a schedule

Nothing is silently lost, and nothing is replayed thirty times. If TRUSTIVAN is down for thirty days, thirty daily occurrences come due at once. The policy is run once, realign, count the rest:
  • One scan runs, not thirty. Thirty identical scans of one image would cost thirty times the registry bandwidth and tell you nothing the last one alone would not.
  • next_run_at advances in whole intervals from the planned time, so the original phase is preserved. A schedule that ran at 02:00 keeps running at 02:00 rather than drifting to whenever the system came back.
  • The skipped occurrences accumulate on the schedule as missed_runs and are shown in the dashboard, because a gap in coverage that nobody is told about is indistinguishable from continuous coverage.
Frequencies are hourly, daily or weekly, with one schedule per asset. There are no cron expressions, so there is no timezone or DST behaviour for an upgrade window to interact with — everything is stored as an absolute instant in UTC, which is precisely what “every 24 hours” needs so it does not become “every 23 or 25 hours” twice a year.

Verifying an upgrade

Then check three things beyond the probes:
  1. A scan completes. Submit one against a small public image and poll it to a terminal state.
  2. database_state on that scan is fresh. If the scanner cache was discarded or egress is blocked, this is where it shows.
  3. trustowl_scheduler_lag_seconds is falling back toward normal. A dead scheduler produces no work, so every other figure looks healthy — see Health and metrics.

Rolling back

Redeploy the previous image. If the newer version applied migrations the older one cannot read, restore the backup rather than rolling the schema back: go run ./cmd/migrate down N reverses the last N migrations and the data in whatever they drop, so it is a development tool rather than an upgrade rollback tool. It refuses to drop the audit trail without --force-drop-audit. This is the practical reason the backup is not optional. Between a schema down that drops a column and a pg_dump taken twenty minutes earlier, only one of them returns the triage decisions that were in that column.

Configuration changes during an upgrade

Two settings are worth re-checking whenever you change versions, because both fail closed and both fail at startup rather than at use:
  • ENV must remain something other than development in a deployment. If a deploy accidentally sets it to development, secrets are regenerated per process instead of being required, and every session is invalidated on each restart.
  • VULN_DB_PROVIDER=mirror requires VULN_DB_REPOSITORIES. Startup fails if it is empty rather than silently reaching for the public internet — which either fails confusingly or, worse, succeeds and defeats the isolation you configured.
See Configuration overview.

What deployment tooling exists

Worth restating here, because an upgrade is when people go looking for it. Compose on a single host is the recommended path, and the gated sequence is infra/scripts/deploy.sh (preflight, backup, migrate, start, health, smoke) with infra/scripts/rollback.sh beside it. Kubernetes manifests exist as a Kustomize base and two overlays, driven by make deploy-demo and make deploy-prod; there is no Helm chart. Terraform provisions a Hetzner cluster optionally and never deploys the application. There is no make deploy-staging and no make deploy-production — those names do not exist. See Topology for the full account of what is and is not included.