eltPulse

Pipelines

A pipeline is a named definition from a source type to a destination type. eltPulse resolves the sync engine automatically — dlt for SaaS/API sources, Sling for database replication — and generates code you can run managed or export to Git.

Connector catalog

The builder exposes 123+ sources and major warehouse destinations. Tier-1 connectors (GitHub, HubSpot, Salesforce, Stripe, Postgres, and more) ship with production codegen, credential forms, and run slice wiring where applicable. Browse the public connector catalog for trust tier, sync mode, and scenario recipes.

Visual canvas

The visual canvas is eltPulse's drag-and-drop pipeline designer — the same pipeline as the form builder, optimized for graph editing:

  • Node graph — stored in sourceConfiguration.canvas; source, transform, and destination nodes with typed edges.
  • Transform inspector — dbt, dlt_dbt, and post_transform sync back into source config on save; includes the workspace dbt project picker.
  • Pulse AI integration — AI can add nodes, wire edges, and patch configs from chat (see Pulse AI).
  • Any warehouse — Snowflake, BigQuery, Redshift, MotherDuck, Postgres, DuckDB, Databricks as destinations on one graph.

Form builder & Quick start

The tabbed builder supports catalog wizard, manual JSON, and Pulse AI create flows. Quick start is the fastest path: destination → source → credentials → run on managed workers.

Artifacts

  • Sync runner — generated Python (dlt) or replication.yaml (Sling).
  • config.yaml — serialized source/destination configuration. Link saved connections (sourceConnectionId / destinationConnectionId); generated YAML may include resolved connection names.
  • eltpulse_workspace.yaml — scheduling, retries, and code location metadata. See Orchestration.

Declarative YAML (GitOps)

Define or update pipelines without the UI via POST /api/elt/pipelines/declaration. The document must include eltpulse_pipeline_declaration: 1 and the same fields as POST /api/elt/pipelines (name, source/destination types, sourceConfiguration, optional dbt, dbtProjectId, _partitionConfig, execution settings).

  • Upsert: upsert: true or ?mode=upsert for idempotent GitHub Actions applies.
  • Content-Type: raw YAML or JSON { "declaration": "..." }.
  • Example: examples/eltpulse-pipeline.declaration.example.yaml.

Runs on (execution targeting)

Each pipeline has Runs on: Inherit (account default), eltPulse-managed, or Customer gateway. Set a default gateway token to route new runs to a named connector. Resolution order and monitor behavior are documented in Concepts; gateway deployment in Gateway.

Edit and delete

Edit regenerates all artifacts on save (and may auto-push to GitHub). Delete removes the pipeline from workspace storage.

Open Pipelines · Open canvas · Concepts · Repositories