Pipeline
The Pipeline screen lets you check how well SingleStore's data ingestion pipelines are running. When a pipeline falls behind or fails, data does not arrive on time. This screen brings together throughput, latency, and errors so that you can quickly pinpoint where the problem is.
What is a pipeline
A SingleStore pipeline automatically pulls data from external sources (files, message queues, and so on) and loads it into tables. In simple terms, it is like a conveyor belt that continuously moves data. If the belt slows down or stops, data piles up, so it is important to watch the processing speed and errors.
Basic screen guide
Select the cluster to query at the top of the screen. The screen consists of a status summary, KPI cards, trend charts, a pipeline details table, and recent errors.
KPI cards
The cards show the key metrics of the overall pipeline.
| Card | Description |
|---|---|
PIPELINES | Total number of pipelines |
FAILING PIPELINES | Number of pipelines that are currently failing |
BATCH SUCCESS RATE | Batch success rate |
FAILED BATCHES | Number of failed batches |
BATCHES | Number of processed batches (batches per interval) |
Each card also shows a secondary unit such as rows/s (avg), batches per interval, or milliseconds (avg).
Trend charts
Two time-series charts show the processing status of the pipelines.
| Chart | Description |
|---|---|
RPS by Pipeline | Rows processed per second for each pipeline (rows per second) |
Batch Success / Failure | Trend of batch successes and failures |
Clicking a specific pipeline in a chart filters both the chart and the table below by that pipeline.
Pipeline Details table
The table shows the processing status of each pipeline. You can search by pipeline, source, or table name in the search box at the top.
| Column | Description |
|---|---|
Pipeline | Pipeline name |
Source | Data source |
Target Table | Target table for ingestion |
Type | Source type |
Kind | Pipeline kind |
RPS (avg) | Rows processed per second (average) |
Rows Total | Total number of rows ingested |
Batches OK | Number of successful batches |
Failed | Number of failed batches |
Queued | Number of queued batches (backlog) |
Clicking a pipeline row in the table opens the batch-level details drawer. In the drawer, you can review the batch metric charts and recent errors for the selected pipeline.
Recent Pipeline Errors
At the bottom of the screen, you can check the pipeline errors that occurred recently.
| Column | Description |
|---|---|
Error Time | Time the error occurred |
Pipeline | Pipeline where the error occurred |
Batch | Batch identifier |
Code | Error code |
Message | Error message |
Host | Host where it occurred |
Instance | Instance where it occurred |
- Opening the batch drawer automatically queries recent errors.
- The error list is shown across multiple pages.
- You can search by pipeline, type, code, or message in the search box.
Query time
You can check the last collection time at the top right of the screen.