Orchestration

eltPulse keeps definitions and orchestration separate: definitions describe what runs (pipelines, code, assets), while orchestration covers when and how work runs — schedules, monitors, run slices, and run policies. We aim to be the enterprise-grade place to run that orchestration end-to-end, while keeping exports and contracts portable so you are never locked in. The same artifacts can be driven by eltPulse or by tools you already use — Airflow, Prefect, GitHub Actions, or a custom runner — because we want customers who choose us, not customers who are stuck.

Schedules and monitors

Schedules are time-based (cron). Monitors react to events (files, webhooks, upstream completions). Both belong in the orchestration layer, not embedded as opaque strings inside EL connectors.

In-product, event triggers are eltPulse monitors. Each monitor has its own Runs on setting (inherit account plane, always eltPulse-managed cron, or always your gateway): when checks are customer-side, your gateway evaluates S3/SQS (or your custom worker) and POSTs results to the control plane; when eltPulse-managed, the cloud cron evaluates them. Monitors link to a saved connection when the monitor type needs cloud API credentials.

Run checks in the app only runs monitors that eltPulse is allowed to evaluate in the cloud; gateway-only monitors are skipped there. See Concepts for how monitors relate to pipelines and connections, and Gateway for the agent API.

Run slices

For backfills and incremental windows, runs can carry a slice key (often exposed as partition_key in generated Python for compatibility). The Run slices page in the app is where you configure that strategy per pipeline.

Where it appears in the product

  • Pipelines — definitions and exports.
  • eltpulse_workspace.yamlscheduling and resilience metadata next to generated code.
  • Runs (signed in) — execution history reported by your runner or CI.
  • Gateway (signed in) — tokens and defaults for self-hosted execution.
  • Gateway (docs) — how the agent process works and how to deploy it.

Roadmap

First-class schedule and sensor experiences in the product, deeper native orchestration, and ongoing interoperability with external engines are tracked on the roadmap. For targeting and data model details, see Concepts.