Runs

A run is one execution of a pipeline: status, timing, optional targeting to a named gateway, and everything your runner reports back — structured logs, telemetry (rows, bytes, progress), and sanitized errors. View and filter runs in the app at Runs.

Structured logs

Each run stores logEntries as a JSON array (capped at 500 lines server-side). Each entry has at (ISO time), level (info, warn, or error), and message (plain text, scrubbed — no raw warehouse credentials).

  • Append one line: PATCH with appendLog: { level, message } — the server sets at to the request time.
  • Replace the whole log: PATCH with logEntries: [...] (same shape as returned by GET).

The same fields work on PATCH /api/elt/runs/:id (signed-in session) and PATCH /api/agent/runs/:id (Bearer gateway token). Managed workers and CI should use whichever auth path matches your deployment.

Telemetry (live + historical)

Runs also store telemetry as JSON with summary (rollup: rows, bytes, progress, phase, resource) and samples (time series, capped at 2000 points). This powers the Runs table columns, dashboard strip, and sparklines on run detail.

  • telemetrySummary — shallow-merge into summary (e.g. cumulative rowsLoaded, bytesLoaded, progress 0–100, currentPhase, currentResource).
  • appendTelemetrySample — append one point (optional at; defaults to now). Prefer calling every few seconds while status is running.
  • telemetrySamples — replace the entire samples array (e.g. post-run backfill).

Webhooks for terminal runs may include telemetrySummary in the JSON body when the run reported metrics. See Webhooks for delivery rules.

Typical PATCH sequence

  1. Create or claim a run (POST /api/elt/runs or poll GET /api/agent/runs).
  2. Set status: "running" and stream appendLog / appendTelemetrySample while work proceeds.
  3. Finish with status: "succeeded" (or failed / cancelled), optional errorSummary, final telemetrySummary, and finishedAt if needed.

Related

  • Concepts — how runs relate to pipelines and gateways.
  • Gateway — Bearer routes including PATCH /api/agent/runs/:id.
  • Pipelines — definitions that runs execute.
  • Security & data — what we persist and what stays in your environment.