eltPulse

Concepts: how the pieces fit

eltPulse splits definitions (what to run) from execution (who runs it). This page maps the main objects and how you steer pipelines and monitors toward eltPulse-managed compute vs a specific gateway.

Diagrams

The first figure is an entity relationship view (what references what in the product). The second is architecture: what runs in eltPulse-hosted environments versus your network, and what crosses the egress-only link (HTTPS + JSON control APIs — not your warehouse bulk traffic).

How pipelines, connections, monitors, runs, and gateways relatePipelines and monitors reference connections by foreign keys. Runs belong to pipelines. The gateway uses the control plane API only.Connectionnamed profile + optional secretsPipelinedefinition (Neon)sourceConnectionIddestinationConnectionId (FK)Runone execution (Neon)instance ofMonitorsensor (Neon)connectionId(FK)enqueues run (pipelineName → pipeline)GatewayBearer = AgentTokenHTTPS to API onlyeltPulse control plane (Next.js API + Neon)persists pipelines, runs, monitors, connections · serves /api/agent/*Gateway never connects to Postgres; it uses JSON APIs over TLS.manifest · runs · connections · heartbeat · monitor report
Pipelines and monitors link to connections by foreign keys. The gateway loads connection payloads through the API, not by talking to Neon.
Where work runs: eltPulse-hosted vs customer gateway, egress-onlyThree columns: your network, the HTTPS control-plane link (TLS, JSON APIs, auth), and eltPulse hosted. No inbound from eltPulse to your VPC. Separate egress to your data plane.YOUR NETWORKVPC · cluster · laptopGateway (long-lived)polls manifest · runs · heartbeatsoptional: monitors · dispatch workersRun workers (optional)ECS / K8s Job / docker — eltPulse connectors, your codeData plane (your accounts)Snowflake · S3 · Postgres…Separate TLS egress from here— not through eltPulseWarehouse bulk stays out of bandTHE GAP — CONTROL PLANE ONLYNothing else crosses this strip · your gateway opens TLS outward1 · TransportHTTPS egressTLS you initiateto eltPulse apporigin only2 · JSON APIsmanifest · runsconnections · hbmonitor reportGET/PATCH whereapplicable3 · Auth & scopeBearer = gatewaytoken onlyNo pipeline codeNo warehouse bulkon this pathNo inbound from eltPulse into your VPCeltPulse never originates connections to your private network for runs or monitorsingestion → warehouse / sources (separate path)ELTPULSE HOSTEDe.g. Vercel + NeonNext.js API/api/agent/* · Clerk · webhooksPostgres (Neon)pipelines · runs · connections…Cloud cron (managed monitors)S3/SQS when Runs on = eltPulse-managedManaged workersfuture — queue consumers foreltpulse_managed runsNo listener required in your network

Reading the diagram: three columns — your network, the control-plane gap (three boxes: transport, JSON APIs, auth/scope), then eltPulse hosted. Arrows at gateway height show the HTTPS path only.

Customer path: only outbound TLS from your side to the app origin for control-plane APIs. eltPulse does not open a path into your VPC for runs or monitors.

eltPulse-managed path: monitors (and eventually workers) run in our environment using connection credentials stored for cloud checks; ingestion may still use env vars in the runner depending on setup.

Core objects

Pipeline
A named definition (source → destination, generated code, workspace YAML). Stored per user. Optional source and destination links are stored as foreign keys (sourceConnectionId, destinationConnectionId) to rows on the Connections page. When set, generated config.yaml may include the connection names for runners that still read source_connection / destination_connection — those keys are derived from the linked row, not authored as the source of truth in JSON. See Pipelines and Connections.
Run
One execution of a pipeline: status, structured logs (logEntries), telemetry (rows, bytes, progress samples), timestamps, triggeredBy (schedule, monitor, webhook, API). Many runs can exist per pipeline over time. View them on Runs; see Runs (docs).
Connection
A saved profile for a source or destination: connector type, non-secret config, and optionally encrypted secrets. Pipelines link optional source/destination profiles by id (builder / API). Monitors pick a connection in the monitor UI, which stores a database connectionId on the monitor row. Gateways can load all profiles (with decrypted secrets) via GET /api/agent/connections. Manage profiles on Connections.
Monitor (sensor)
Watches an external signal (e.g. S3 object count) and can enqueue a run of a named pipeline when a threshold is met. Optional link to a connection for credentials. Configure on Monitors.
Gateway (named token)
A named connector you create on the Gateway page: one row in AgentToken with its own Bearer secret. Your process uses that secret on /api/agent/*. Org-scoped tokens belong to a workspace; personal tokens belong to your user.

Relationships (quick)

  • Pipeline → runs: triggers (schedule, monitor, webhook, API) create runs pointing at that pipeline's pipelineId.
  • Pipeline → connection: optional foreign keys sourceConnectionId / destinationConnectionId to saved profiles (same idea as connectionId on monitors). Exported YAML may still surface resolved names for runners.
  • Monitor → pipeline: each monitor names a pipelineName (must match an existing pipeline). It does not embed pipeline code.
  • Monitor → connection: cloud monitor types store a foreign key to a connection so checks use that profile's connector and secrets (e.g. S3 connector for an S3 file-count monitor).
  • Gateway → connections: a process using a gateway token may call GET /api/agent/connections and receive decrypted secret keys for that user's connections (use only in trusted runtime). See Security & data.
  • Pipeline definition vs runner env: even when a pipeline links a saved connection, many runners still expect secrets as environment variables where ingestion executes (CI, gateway host, or managed worker), while the Connections page documents which keys to set. If the gateway loads secrets via GET /api/agent/connections, it can align with the same profile for agent-driven flows.

Account execution plane

Your user (or org context) has an execution plane setting: eltPulse-managed vs your infrastructure (customer). It affects the default when objects use inherit — it does not delete pipelines; it changes where new runs are expected to execute whenRuns on is set to inherit. Switch it on Gateway.

Pipelines: where new runs execute

Each pipeline has Runs on (executionHost):

  • Inherit — follow account execution plane: managed plane → eltPulse-managed workers; customer plane → customer gateway path (see resolution below).
  • eltPulse-managed — force new runs to eltPulse-operated execution (telemetry still in Neon).
  • Customer gateway — force customer path: runs get a targetAgentTokenId from your defaults unless overridden per run.

In the builder, you can also set the pipeline's default gateway (named token). That pins new runs to that connector when the run doesn't specify another target and the resolver needs a gateway.

Who picks targetAgentTokenId on a new run?

For customer-gateway paths, the control plane resolves in order:

  1. Explicit per run — if the API body includes targetAgentTokenId (named token id), that gateway (if you own it) wins. Use null to mean "any gateway" with this user's token (untargeted).
  2. Pipeline default — the pipeline's default gateway, if set and still valid.
  3. Org default — when working in an org context, the organization's default gateway token.
  4. Account default — your user's default gateway on the Gateway page.
  5. Single token in scope — if you have exactly one personal or org-scoped named token in that context, it may auto-pin unrouted runs.

A named gateway polling with its Bearer token only receives runs where targetAgentTokenId is null (any gateway) or equals that token's id. Another named gateway cannot claim a run targeted at a different token.

Monitors: where checks run

Each monitor has its own Runs on (execution_host in the API): inherit, eltPulse-managed, or customer gateway, with the same meaning as pipelines relative to your account execution plane.

  • eltPulse-managed — eltPulse's cloud cron evaluates the monitor (S3/SQS today); your gateway must not POST results for it.
  • Customer gateway — only your gateway should evaluate and call POST /api/agent/monitors/:id/report.
  • Inherit + customer plane — checks run on the gateway; inherit + managed plane — checks run in eltPulse cron.

Monitors reuse the same connections as the rest of the app so credentials stay in one place.

Schedules

Cron / interval definitions live in workspace YAML and the Schedule experience. When a schedule fires, it creates a run; that run still follows the pipeline's Runs on and gateway resolution above.

Related guides