Deploy and upgrade

All commands run from a checkout of vantage-deployment on your workstation. CLUSTER_ENV picks the cluster (inventory inventory/<env>.yml and secrets directory pki/<env>/); --env picks prod or preview inside it.

First deploy on GCP or VultrCopy Anchor Link

1. Install the tools: ./scripts/prereqs.sh.

2. Create the state and backup buckets once, from a B2 master key:

   Copy Text

   ./scripts/b2-buckets.sh <master key file> ~/.b2 vantage-tfstate vantage-backups
   export B2_KEY_ID="$(sed -n 1p ~/.b2/vantage-tfstate.key)" \
          B2_APPLICATION_KEY="$(sed -n 2p ~/.b2/vantage-tfstate.key)"

3. Create the servers. On Vultr also export VULTR_API_KEY=….

   Copy Text

   cd terraform/gcp                                 # or terraform/vultr
   cp terraform.tfvars.example terraform.tfvars     # project or region, domain, email, ssh keys
   cp backend.hcl.example backend.hcl
   tofu init -backend-config=backend.hcl \
     -backend-config="access_key=$B2_KEY_ID" -backend-config="secret_key=$B2_APPLICATION_KEY"
   tofu apply                                       # writes ../../inventory/<cluster_env>.yml
   tofu output -raw dns_records                     # Vultr: add these at your DNS provider
   cd ../..

4. Wait until every name resolves to the workers (dig +short analytics.<domain>).

5. Put a GitHub token with read:packages in pki/prod/ghcr-pull.token, then fill scripts/deploy.vars from scripts/deploy.vars.example: GHCR_USER, VERSION_VANTAGE and the three PLATFORM_ADMIN_* values.

6. Provision the platform:

   Copy Text

   CLUSTER_ENV=prod ansible-playbook -i inventory/prod.yml playbooks/site.yml

   site.yml runs, in order: bootstrap, consul, vault, nomad, caddy, database, nomad_vault (generated secrets, Vault roles, placeholders), ingress (hostnames into Consul and the nomad/jobs variable).

7. Deploy the jobs:

   Copy Text

   CLUSTER_ENV=prod ./scripts/deploy.sh

   It pre-pulls every image on every worker, submits the platform jobs (zitadel, zitadel-login, temporal, rustfs, rustfs-bootstrap, oauth2-proxy, backup), then the analytics jobs and observability, publishes the ingress names, runs provision-zitadel.sh and waits until every public name serves.

8. Check it: CLUSTER_ENV=prod ./scripts/verify.sh. It passes only when every allocation is running and every public name answers its expected code.

9. Sign in at https://analytics.<domain> as the platform admin with user_password from secret/zitadel/app, and change the password.

10. Fill the placeholders: SMTP, the backup bucket, and the optional AI, digest and billing keys (below).

deploy.sh flags: --env prod|preview|both, --stack analytics|observability|all, --tag <tag> (one tag for every first-party job), --fresh (discard allocations first). scripts/status.sh -w shows node capacity and every job, refreshed every 5 seconds.

Your own serversCopy Anchor Link

Write inventory/prod.yml from inventory/prod.yml.example (hosts, node_ip, groups, nomad_volumes per worker, cluster_env_expected: prod), point DNS at the workers, then start at step 5.

Add previewCopy Anchor Link

1. tofu apply again (adds the three preview names); add them at your DNS provider if it is not Cloud DNS.
2. CLUSTER_ENV=prod ansible-playbook -i inventory/prod.yml playbooks/site.yml.
3. ./scripts/deploy.sh --env prod (creates preview's bucket, callback and hostnames).
4. Export PLATFORM_ADMIN_EMAIL, PLATFORM_ADMIN_FIRST_NAME, PLATFORM_ADMIN_LAST_NAME, then ./scripts/deploy.sh --env preview.

./scripts/reset-preview.sh --dry-run lists, and ./scripts/reset-preview.sh removes, every preview job, database, warehouse database and user, key and bucket object, without touching prod.

UpgradeCopy Anchor Link

Releases are image tags published by the vantage repository's CI. Preview runs a version before prod.

1. Set VERSION_VANTAGE in scripts/deploy.vars.preview to the new tag.
2. ./scripts/deploy.sh --env preview, then ./scripts/verify.sh --env preview.
3. Copy VERSION_VANTAGE into scripts/deploy.vars.
4. ./scripts/deploy.sh --env prod, then ./scripts/verify.sh --env prod.

The control-plane schema migrates on start: the vantage-web image runs bootstrap-controlplane.sh (migrations, roles, row-security policies, under a Postgres advisory lock) before the server. analytics-web rolls out one canary and promotes it automatically; a canary that is not healthy within 5 minutes is reverted. To hold one job on an older release, set TAG_<JOB> in deploy.vars (for example TAG_DATAPLANE=v0.1.0).

Never run drizzle-kit migrate or push by hand: they can recreate tables without their row-security policies. Use the bootstrap script, which applies both.

Change a secretCopy Anchor Link

Configuration is read from Vault when a job renders its template. A change is a patch and a restart, never a new image.

export CLUSTER_ENV=prod
source scripts/lib.sh
nomad_env
export VAULT_ADDR="https://$(resolve_ip vault_servers 0):8200" \
       VAULT_CACERT="$PKI_DIR/ca.crt" \
       VAULT_TOKEN="$(jq -r .root_token "$PKI_DIR/vault-init.json")"

vault kv patch secret/zitadel/smtp host=<relay> port=587 user=<user> password=<password> from=<sender>
nomad job restart zitadel

vault kv patch secret/vantage/anthropic api_key=<key>
nomad job restart analytics-web

Use patch, not put (a put drops the fields you did not type), and nomad job restart, not run (an unchanged spec places no new allocation, so the template is not re-read).

Back up and restoreCopy Anchor Link

The backup job runs at 02:30 UTC.

nomad job periodic force backup     # run one now
nomad job status backup

StarRocks is not backed up. Tables from a scheduled source come back by rewinding the source and letting it reload; pack views by re-installing the pack. Uploaded files are in the staging mirror, but nothing reloads them, and rows posted to load-batch exist nowhere else.

Restore on a self-hosted Postgres, all four databases from the same date:

nomad job stop analytics-web dataplane zitadel zitadel-login temporal
ssh <database host> 'sudo -u postgres pg_restore -d analytics --clean --if-exists /path/to/<date>/analytics.dump'
# repeat for zitadel, temporal, temporal_visibility
./scripts/deploy.sh

On GCP restore Cloud SQL from its own backups or a point in time.

Local composeCopy Anchor Link

From vantage-deployment/compose, with the vantage repository next to it:

./vantage.sh up          # data and identity services; builds the dataplane
./vantage.sh provision   # applies vantage-config.json, writes env/vantage-analytics.env, starts analytics-web
./vantage.sh status
./vantage.sh down        # stop; down -v also deletes the data and the instance keys

Overrides go in compose/env/vantage.env, created from vantage.env.example on first run. Open http://localhost:4445/signup to create a workspace; the invitation lands in Mailpit. There is no /api/health; /signup answering 200 and the dataplane's /healthz are the liveness checks.

To restart only the app later, vantage.sh must have set its environment first:

export VANTAGE_DEPLOYMENT_DIR=$(cd .. && pwd) VANTAGE_DIR=$(cd ../../vantage && pwd) \
  ZITADEL_MACHINEKEY_DIR=$PWD/zitadel/vantage-machinekey
docker compose --env-file env/vantage.env -f docker-compose.yml up -d analytics-web

NextCopy Anchor Link

• Configuration reference
• Tenant isolation