Service tokens
How do machine callers authenticate?
Machine callers authenticate with static bearer tokens set in environment variables, one set per caller. They are operator credentials, set in Vault and rendered into the jobs, with no UI and no per-workspace issuance. People and AI agents use other credentials: a session cookie, or a personal MCP token.
A service token carries no workspace. Every
/api/internal/*request names itstenantId(or a database) in the body, and the dataplane acts on whatever it names. A holder of anyINGEST_TOKENS_<NAME>value can load into, provision users on, or tear down any workspace's warehouse. Keep these tokens inside the cluster and never give one to a customer.
Tokens the dataplane accepts
Any variable named INGEST_TOKENS_<NAME> defines a caller called <name>; adding a caller is
a secret and a restart, not a release. The value is a comma-separated current[,previous]
list, compared in constant time.
| Variable | Caller | Set on the cluster |
|---|---|---|
INGEST_TOKENS_WEB | analytics-web, presenting INGEST_TOKEN | yes, ingest_token_web |
INGEST_TOKENS_AIRBYTE | an Airbyte deployment | yes, ingest_token_airbyte |
INGEST_TOKENS_OPENFN | an OpenFN deployment | yes, ingest_token_openfn |
INGEST_TOKENS_N8N, INGEST_TOKENS_BACKEND | n8n, norims-backend | Norims clusters |
All three seeded values live in Vault at secret/analytics/app (preview:
secret/preview/analytics/app). In production the dataplane refuses to start with no
INGEST_TOKENS_* set.
The name only labels the caller in logs; every caller may call every endpoint.
Rotate a token
- Prepend the new value:
vault kv patch secret/analytics/app ingest_token_airbyte=<new>,<old>andnomad job restart dataplane. - Move the caller to
<new>. - Patch again with
<new>alone and restart.
For ingest_token_web the app presents the whole field as INGEST_TOKEN, so rotate it with a
single value and restart both jobs together.
Endpoints on the dataplane
Reached inside the cluster at the dataplane service, or at https://uploads.<domain>/api/internal/*,
which the ingress exempts from operator sign-in. All take and return JSON.
| Endpoint | Body | Does |
|---|---|---|
POST /api/internal/load-batch | tenantId, sourceId, streamName, rows[], maxCursor?, runId? (64 MB) | Creates or widens the raw table, stream-loads rows, registers the dataset, advances the cursor; returns { rowsLoaded } |
GET /api/internal/cdc-plan?tenantId&sourceId | What the next sync should read: sourceTable, cursorField, streamName, cursorValue, tenantPredicate, entityType | |
POST /api/internal/streams | tenantId | The workspace's streams and their cursors |
POST /api/internal/schedule, /schedule/pause, /schedule/trigger, /schedule/delete | tenantId, sourceId, streamName, type, plus cron / paused | Temporal schedule for a stream |
POST /api/internal/reset-stream | tenantId, sourceId, streamName, clear? | Rewinds the cursor; with clear: true drops the raw table |
POST /api/internal/ensure-database, list-tables, list-tables-detailed, table-columns, sample-json | database and table names | Warehouse reads and database creation |
POST /api/internal/provision-dbt-user, revoke-dbt-user, provision-readonly-user, readonly-grants | database, user, password | Warehouse users for dbt and read-only access |
POST /api/internal/create-mv, refresh-mv, drop-warehouse-objects, teardown-warehouse | Modeled datasets, sample cleanup, workspace teardown |
load-batch, cdc-plan and reset-stream accept only sources of type n8n; any other
answers 404 Unknown or non-n8n data source. Sources created as Airbyte or OpenFN in
Integrations cannot load through load-batch.
load-batch
curl -s https://uploads.example.com/api/internal/load-batch \
-H "Authorization: Bearer $INGEST_TOKEN" -H 'Content-Type: application/json' \
-d '{
"tenantId": "<workspace id>", "sourceId": "<source id>", "streamName": "orders",
"rows": [{ "id": 1, "facility": "Westlands Store", "revenue": "1250.00", "event_offset": 1042 }],
"maxCursor": 1042
}'| Rule | Detail |
|---|---|
| Key | the source's keyColumn (default id), not null in every row |
| Columns | [A-Za-z_][A-Za-z0-9_]*; tenant_id, _ingested_at, _source_id are set by the dataplane |
| Table | tenant_<slug>.raw_<datasetSlug or source name slug> |
maxCursor | numeric; stored as the stream's cursor for the next cdc-plan |
| Failure | 400 with message; the source turns error and runId's run is marked failed |
| Plan limits | not checked here; on a standalone Vantage, scheduled syncs ask analytics-web (loads-allowed) before they run and skip when the plan refuses |
Tokens analytics-web accepts
| Variable | Caller | Routes |
|---|---|---|
CUBE_SERVICE_TOKEN | Cube | /api/internal/cube/schema, schema-version, tenants, creds |
INGEST_TOKEN (the dataplane presents the first INGEST_TOKENS_WEB) | dataplane | /api/internal/packs/dataset-ready, /api/internal/billing/loads-allowed, /api/internal/ai/digests |
PROVISION_TOKENS_BACKEND | norims-backend | /api/internal/provision-tenant, provision-source, teardown-tenant, /api/packs/validate |
See Configuration reference for defaults and which are required.