What is a pipeline?

A pipeline is the automated job that moves data from one source connection into your warehouse. Each pipeline pulls data from a single source (for example your Meta Ads account or your MEWS property), transforms it, and writes it to your destination so it is ready for Data Products, metrics, and dashboards.

Every source connection you add gets its own pipeline. The Pipelines page lists all of them in one place and lets you check their status, trigger runs, review history, and change run intervals.

Where do you find the Pipelines?

To open it, go to Engineering → Processing → Pipelines in the left sidebar. At the top you see a summary of your space: how many pipelines are running, failing, or degraded, and the total count. Below it sits the list of every pipeline in your space.

Each row shows:

From here you can:

• Search: type in the Search pipelines field to jump to a pipeline by name.

• Filter: use the All, Running, Failing, Degraded, or Idle tabs to narrow the list by state.

• Add: click + New pipeline (top right) to create a new pipeline.

• Open: click any pipeline name to see its detail view.

Reading pipeline status and health

Each pipeline shows two separate signals.

• Status tells you whether the pipeline runs on its schedule. Common values are Live or Active (running on schedule), Running (a sync is in progress right now), and Paused (no automatic runs).

• Health tells you whether the recent runs worked. Healthy means the latest runs went through. Failing means one or more runs hit an error and the pipeline might need attention.

A pipeline can be Paused and Healthy, or Live and Failing, so the two badges do not always match.

Opening a pipeline

Click a pipeline name to open its detail view. At the top you see the pipeline name, its status and health badges, and the key facts: whether it runs automatically, when it was created, when it last ran, and when the next run is due. The Auto reload toggle keeps the view refreshing while a sync is running, and the Options button (top right) holds the pipeline actions.

Four cards summarise recent activity:

• Total Syncs: how many sync runs the pipeline has completed.

• Success Rate: the share of recent runs that succeeded.

• Throughput (7d): rows processed in the last 7 days.

• Data Volume (7d): how much data was transformed in the last 7 days.

Below the header, the detail view is split into tabs.

Flow

Flow is the default tab. A diagram at the top shows the data path: Source → Direct Sync → Destination, and the sync step carries its own health badge, for example Healthy.

Two shortcuts sit right on the diagram:

• Hover over the Source card and click Go to connection to open the source connection, for example to reconnect it.

• Under the sync step, click Pause to stop scheduled runs or Run now to start a sync immediately.

Below the diagram, four panels give you the details:

• Configuration: created date, last updated, and estimated token cost.

• Resource IDs: the Pipeline, Source conn., and Source integ. IDs. The Go to link next to the source connection opens it too.

• Schedule: the run Frequency and the next Upcoming runs.

• Performance: total syncs, plus Last 7 days and Last 30 days figures (syncs, errors, error rate, rows synced, data volume), and the current Status (Execution and Health).

State

Lists every stream (table) the pipeline syncs, with the time each was last updated. Expand a stream to see its sync state: the internal markers Indicate uses to know where the last run stopped, such as last_run and next_run. You can select individual streams or use Delete all state to reset them.

The _absolute_start_at / _absolute_end_at timestamps.

• Absolute start at and Absolute end at show the first and last day we pull data from the source for this connector.

• For some connectors this historical range is predefined (limited lookback). For others we can pull all available historical data, and the timestamp reflects that full range.

Resetting state re-syncs data. Deleting a stream's state makes the next run start from scratch for that stream. Only reset state when you want a clean re-sync, and check Run History first to understand why a run failed.

Run History

A searchable log of every run. Filter by result (All, Succeeded, Failed, Skipped) or by stream, and open any entry to see exactly what happened. This is the first place to look when a pipeline is Failing, because here the cause for the failure is also visible as provided by the data source.

Full run history is a Pro feature. On the Core plan the Run History shows only the most recent two days. Upgrade to Pro to access the a more detailed run history.

Schema Log

Shows the history of schema changes the pipeline has detected in your source over time (for example new, changed, or removed fields). Use it to understand when the structure of your incoming data changed.

Insights

Shows the insights generated for this pipeline. You can also find them under Engineering → Semantic → Insights. They include the KPIs for that source.

Settings

Holds three sections:

• General (rename the pipeline),

• Schedule (how often the pipeline runs),

• and the Danger zone (truncate the pipeline data). Each is covered step by step below.

Pause or resume a pipeline

1. Open the pipeline.

2. Click ⚙️ Options in the top right.

3. Choose Pause pipeline to stop automatic runs, or Resume pipeline to start them again.

Pausing keeps all data and configuration. It only stops scheduled runs.

Run a pipeline now (Pro)

1. Open the pipeline.

2. Click ⚙️ Options→ Run now.

3. The pipeline starts a sync immediately instead of waiting for the next scheduled run.

Fix a Failing pipeline (Force healthy)

When a pipeline shows Failing but you have already resolved the underlying cause (for example fixed the source credentials or a setting), you can clear the error and reset the status to Healthy.

1. Open the failing pipeline.

2. Click ⚙️ Options.

3. Click Force healthy.

4. The flow diagram updates and the health badge turns Healthy.

5. Open ⚙️ Options again and choose Run now to confirm the pipeline runs cleanly. Run now is a Pro feature. If you have a Core space the pipeline will sync automatically over night.

Force healthy only overrides the status. It does not fix the root cause. If the underlying problem is still there, the next run will fail again. Always review Run History first to find and fix the real error.

Rename a pipeline

1. Open the pipeline and go to the Settings tab.

2. Under General, click the Name field and type a new name.

3. Click Save.

Change the sync schedule (Pro)

1. Open the pipeline and go to the Settings tab.

2. Under Schedule, set the Frequency: choose Every how many Days or Hours.

3. Click Update schedule.

Delete or truncate a pipeline

Both destructive actions live in Settings → Danger zone (and under ⚙️ Options → Delete pipeline). Truncating clears the synced data but keeps the pipeline; deleting removes the pipeline entirely. Because both are permanent, they are covered in their own guide: see Delete a pipeline.

Troubleshooting

My pipeline shows Failing.

• Open the pipeline and go to Run History. Filter by Failed and open the entry to see the error. Fix the cause in your source system or connection, then run the pipeline again.

I clicked Force healthy but the next run failed again.

• Force healthy only overrides the status, it does not fix the problem. Review Run History, resolve the underlying error, and then run the pipeline again.

Run now is greyed out or shows a Pro label.

• Manual pipeline runs require the Pro plan. On Core, the pipeline still runs automatically on its schedule. Upgrade to Pro to trigger runs manually.

I cannot change the schedule.

• Schedule configuration is a Pro feature. On Core, the Schedule section is locked. Upgrade to Pro to set custom intervals and operating hours. Some sources sync only once a day by default, so increasing the sync schedule won’t change the output in those cases.

My data looks incomplete after a run.

• Check Run History for skipped or failed streams, and check the State tab to confirm each stream updated.

Some Options entries are missing.

• The Options menu changes with the pipeline state (a paused pipeline shows Resume, a live one shows Pause). Destructive actions also require admin rights. Contact your workspace admin if you cannot see them or use them.