eltPulse

Run slices

Run slices let you scope a pipeline run to a date or key window — the same operational pattern Fivetran and Airbyte use for backfills, without giving up ownership of generated code. In eltPulse, a slice value is passed as partition_key into generated dlt or Sling pipelines.

When to use slices

  • Backfill — replay historical days or tenant keys after a schema change or outage.
  • Bounded incremental — run one day at a time for large SaaS sources instead of one giant sync.
  • Monitor triggers — eltPulse monitors can queue one run per slice value when a file lands or a schedule fires (see Orchestration).

Configure in the app

  1. Open Run slices in the sidebar (or from a pipeline row).
  2. Pick a pipeline and set the partition column (e.g. date, updated_at) and strategy.
  3. When you start a run, pass a slice value — or let monitors supply partition_values per trigger.

The builder and run-slices page show a capability badge per source: date/key slice wired, full replace only, or not yet wired. Tier-1 connectors in the catalog are held to a high bar — either honest slice support or an explicit "full replace only" label.

How codegen uses partition_key

eltPulse picks the right mechanism per connector family:

  • dlt SaaS — start_date, since, or env-bound incremental cursors (e.g. HubSpot, Salesforce, Personio).
  • Sling database replication — update_key + incremental mode on the stream.
  • Filesystem / S3 — path prefix filters for Hive-style layouts.
  • REST API — query params such as since or cursor.

Verified sources with production slice wiring are tracked in codegen; the UI reads the same registry so labels stay in sync with generated Python.

Declarative YAML and API

Pipeline declarations accept _partitionConfig in sourceConfiguration — same shape as the run-slices editor. See Pipelines for POST /api/elt/pipelines/declaration.

Related

  • Connectors — trust tiers and which sources support slices today.
  • Runs — telemetry and logs per slice run.
  • Connector catalog — browse 123+ sources with sync mode labels.