Pipelines

A pipeline is a named definition from a source type to a destination type. eltPulse resolves the right sync engine automatically based on your source and destination combination.

Catalog

The builder exposes a broad catalog of sources and destinations. Generators are implemented incrementally: GitHub and REST are structured in the UI; other sources use a generic template until parity with the original Python builder lands.

Visual canvas

The canvas edits the same pipeline as the form builder — including sourceConfiguration.canvas for the node graph and transform inspector fields. Transform nodes sync dlt_dbt / post_transform back into source config on save. The canvas transform inspector includes the same workspace dbt project picker as the builder.

AI Builder

Describe a pipeline in natural language from the builder create tab or floating widget. The AI generates savePayload compatible with POST /api/elt/pipelines, including optional post-load dbt and dbtProjectId. See AI Builder.

Artifacts

• Sync runner — the executable artifact eltPulse generates for your pipeline.
• config.yaml — Serialized source/destination configuration for reproducibility. Link saved connections from the builder (stored as sourceConnectionId / destinationConnectionId); generated YAML may include resolved source_connection / destination_connection names when those links exist (see Concepts).
• eltpulse_workspace.yaml — Workspace metadata (scheduling, retries, code location) for your repo layout. See Orchestration for how schedules relate to pipeline definitions.

Declarative YAML (GitOps)

You can define or update a pipeline without the visual builder by posting a **YAML declaration** to POST /api/elt/pipelines/declaration (same session auth as the rest of the app). The document must include eltpulse_pipeline_declaration: 1 and the same fields as POST /api/elt/pipelines (e.g. name, sourceType, destinationType, sourceConfiguration, optional dbt transform config, dbtProjectId, _partitionConfig, execution settings).

• Upsert: set upsert: true in YAML, or call with query ?mode=upsert, to create or **replace** the pipeline with the same name and resolved tool (idempotent applies from GitHub Actions).
• Content-Type: send raw YAML as application/yaml, or JSON { "declaration": "..." } if your client prefers JSON wrapping.
• Example file in the product repo: examples/eltpulse-pipeline.declaration.example.yaml.

After the control plane stores the definition, your **gateway** polls GET /api/agent/runs and receives pipelineCode, configYaml, and workspaceYaml — the same as pipelines created in the UI.

Edit and delete

Use Edit on a row to change definition; we regenerate all artifacts on save. Delete removes the row from your workspace storage.

Where runs execute (Runs on)

In the builder, each pipeline has Runs on: Inherit (follow your account execution plane), eltPulse-managed, or Customer gateway. You can also set a default gateway (named token) so new runs route to that connector unless a run is created with an explicit override.

For the full resolution order (per-run override → pipeline default → org default → account default → single-token auto-pin) and how monitors use the same ideas, read Concepts. For deploying the process that polls runs, see Gateway.

Open Pipelines · Concepts · Repositories layout