5.2. Distributed tracing
Explore all trace data and analyse how requests flow between services.
Overview
The Distributed tracing screen visualises the whole path a request takes through several services in a microservices environment. A trace is a record of the path one request took through the several services of a distributed system.
A heatmap gives you the distribution of the trace data at a glance, so you can identify slow requests and failed requests quickly. Drill down into a selected region to reach the span details of an individual trace.
Open it from the Distributed tracing menu in the left sidebar.
Note: distributed trace data is collected from applications instrumented with the OpenTelemetry SDK. If you have not connected it yet, click the OpenTelemetry integration button at the top of the screen to see how.
Screen layout
| Area | Description |
|---|---|
| Top header | The OpenTelemetry integration button |
| Heatmap | A distribution chart of time (X axis) against response time (Y axis). Drag to select a range |
| Tabs | Overview / traces / error causes / response time explorer / attribute comparison |
| Filter panel | Sets filter conditions such as service name, span name and trace ID |
| Chart selection | Shows the time and response time range selected on the heatmap |
| Baseline | On the attribute comparison and response time explorer tabs, shows the comparison baseline outside the selection |
| Options | The setting for excluding auxiliary requests |
| Content area | The analysis results for the selected tab |
Main features
How to read the heatmap
The heatmap visualises the distribution of time against response time as colour density. The horizontal axis (X) is time and the vertical axis (Y) is the trace's response time. Each cell's colour represents the density of requests in that time and response time range.
- The darker the colour, the more requests occurred in that response time range at that time.
- Colour concentrated towards the top means many requests with long response times.
- Red points at a particular time mean errors occurred then.
- In a healthy state, most points sit towards the bottom (low response times).
A colour legend sits at the top left of the heatmap. Its left end (light) marks regions with few requests, its right end (dark) regions with many.
Where a response time SLO target is configured, the heatmap draws the target threshold as a line. Requests above that line have passed the SLO target.
Click the expand button to the right of the heatmap to see it larger, in full-screen mode.
Selecting a region on the heatmap
Drag on the heatmap to select a particular time and response time range. Only the trace data within it is then shown in the tabs.
- Click and drag across the region you want to investigate.
- The selection appears on the Chart selection row as time range and response time range tags.
- Where you selected an error region, an Error tag appears too.
- The tab contents refresh automatically against the selection.
- To cancel the selection, click Clear selection on the Chart selection row.
Tip: select the region where response times spiked (the cluster of points at the top of the heatmap) and then look at the Traces or Error causes tab to find the cause quickly.
Analysis by tab
Overview tab
Summarises all the trace data.
The summary cards at the top carry the following.
| Metric | Description |
|---|---|
| Services | How many distinct services appear in the traces |
| Requests | Total requests per second (/s) |
| Errors | Errors as a proportion of all requests (%) |
| p50 / p95 / p99 | Response time percentiles (milliseconds) |
The table below shows the request count, error rate and response time percentiles per root service and root span name.
Note: this table shows root spans only. A root span is the first span of a trace, where the request begins.
You can do the following from the table.
- Click a service name (root service): opens the span waterfall chart dialog for a sample trace of that service.
- Click a span name: goes to the Traces tab, filtered to that service and span.
- Click an error rate: goes to the Error causes tab for that service and span.
- The filter icon: applies a filter for that span.
- The trace icon: opens the span waterfall chart for a sample trace.
Note: the root service is the service a request first enters. p50/p95/p99 mean that 50%, 95% and 99% of all requests respectively were handled within that response time.
Traces tab
A table of the individual traces found.
| Column | Description |
|---|---|
| Trace ID | The trace's unique identifier (first 8 characters shown) |
| Root service | The name of the service the request first entered |
| Name | The root span's operation name |
| Status | OK or Error |
| Response time | How long the whole trace took (milliseconds), with a bar showing the relative size |
Click a trace ID or name to open that trace's span waterfall chart in a dialog.
Note: where the result is large, a message about the maximum row limit may appear. Select a narrower region on the heatmap, or add a filter, to reduce the scope.
Error causes tab
Analyses the spans where errors occurred among the traces in the selected range.
Rather than simply listing the traces containing errors, this tab highlights which span of which service the error originated in.
| Column | Description |
|---|---|
| Service name | The service where the error occurred |
| Span | The name and labels of the span where the error occurred |
| Error | The error message |
| Sample trace | The ID of a sample trace containing that error (click to open the span details) |
| Percentage | That error's share of all requests |
Click a Sample trace link to see the span waterfall chart of an actual trace containing that error.
Response time explorer tab
Visualises the response time distribution of the selected traces as a flame graph.
A flame graph visualises the call stack and the time spent in each frame. The wider a block, the more time was spent in that span.
- Click a block to open the span waterfall chart of a sample trace containing that span.
- Selecting a region on the heatmap first and then looking at this tab helps you find which span is the bottleneck among the selected requests.
Attribute comparison tab
Compares the attribute distribution of the traces you selected on the heatmap (the selection) with that of the other traces in the same time window (the baseline).
Where the selection carries a higher proportion of some attribute value than usual, that attribute may be related to the cause of the problem.
Each attribute appears as a card, inside which the baseline and selection proportions are compared as bars, per attribute value.
Tip: select the slow period on the heatmap and look at the Attribute comparison tab to identify unusual attributes quickly — a particular HTTP status code, user agent or database table.
Click an attribute value on a card to see the span waterfall chart of a sample trace carrying that attribute.
Using filters
Add a filter to see only the traces matching a particular service, span or trace ID.
To add a filter:
-
Click Add filter on the filter row.
-
Choose what to filter on from the Field dropdown.
Field Description Root service name The name of the service the request first entered Root span name The root span's operation name Trace ID The unique ID of a particular trace -
Choose an operator.
Operator Meaning =Equals !=Does not equal ~Contains !~Does not contain -
Enter a value.
-
Click the check button to apply the filter.
Applied filters appear as chips. Click a chip to edit it, or its X button to delete that filter. To remove every filter at once, click Clear all.
Note: adding several filters returns only the traces that satisfy all of them.
Option: excluding auxiliary requests
The Options row of the filter panel carries the Exclude auxiliary requests such as monitoring and the Kubernetes control plane checkbox.
Turn it on (the default) and auxiliary requests — internal Kubernetes management calls, monitoring health checks — are excluded from the analysis. Leave it on when you want to analyse real user requests only.
Trace details: the span waterfall chart
Click a trace ID or name in the trace list and that trace's span waterfall chart opens in a dialog.
The span waterfall chart visualises the order in which a request passed through each service and how long each step took.
A summary of the trace appears at the top of the dialog.
| Item | Description |
|---|---|
| Start time | When the trace began |
| Response time | How long the whole trace took (milliseconds) |
| Status | OK or Error |
The chart is split into a service and operation area on the left and a time axis area on the right.
- A horizontal bar starts where the span started, and its length is how long the span took.
- Each span's service name gets its own colour.
- Child spans are indented to express the hierarchy. Click a parent span's name to collapse or expand its children.
- Spans where an error occurred carry an error icon.
- Switch between Show subtrace and Show full trace to see just one service's child spans or the whole trace.
Hover over a span and a tooltip gives the service name, operation name, response time, status, child span count and type (HTTP, gRPC, Kafka and so on).
Looking at span details
Click a span's bar, or its info button, and the span's detail dialog appears.
| Item | Description |
|---|---|
| Name | The span's operation name |
| Service | The service the span belongs to |
| Start time | When the span started |
| Response time | How long the span took (milliseconds) |
| Status | OK or Error, with the message |
| Span ID | The span's unique identifier (copyable) |
| Details | Protocol-specific details (an SQL query, an HTTP request). Syntax-highlighted and copyable. |
| Attributes | The attributes attached to the span (HTTP URL, database query and so on). Each value can be copied individually. |
| Events | Events that occurred while the span ran (an exception stack, for example). Each event's time is given as the elapsed time since the span started. |
The span type is shown as a badge. The supported types are as follows.
| Type | Description |
|---|---|
| HTTP | An HTTP request and response |
| gRPC | A gRPC call |
| Kafka | Kafka message handling |
| Redis | A Redis command |
| Mongo | A MongoDB query |
| PostgreSQL | A PostgreSQL query |
| MySQL | A MySQL query |
| CH | A ClickHouse query |
| ZK | A ZooKeeper operation |
| MC | A Memcached operation |
OpenTelemetry integration
Click the OpenTelemetry integration button in the header at the top of the screen to open the application instrumentation guide dialog.
OpenTelemetry is an open-source observability standard for collecting traces, metrics and logs. Its SDKs, available for Java, Python, Go, Node.js, .NET and other languages, let application code send trace data to OPENMARU Observability.
The dialog carries the following.
- OPENMARU Observability URL: the endpoint address to send trace data to (copyable)
- API key: the key needed for authentication (copyable)
- Per-SDK integration guide: the configuration code and environment variables for the language you use
Note: eBPF-based automatic tracing is supported too. Trace data collected without code changes appears on the Distributed tracing tab of the application details. This screen (the Distributed tracing menu) shows mainly OpenTelemetry SDK instrumentation data.
Worked examples
Analysing the cause of slow requests
- On the Distributed tracing screen, find the period where points are concentrated at the top of the heatmap (the high response time region).
- Drag to select that region.
- On the Overview tab, see which service has the high response time.
- On the Traces tab, click an individual trace to see its span waterfall chart.
- Click the widest bar on the span waterfall chart (the longest span) to see its detailed attributes.
- On the Attribute comparison tab, find what the slow requests have in common — a particular endpoint, a database query.
Tracking down the cause of errors
- Select the region showing errors (the red points) on the heatmap.
- On the Error causes tab, see which service and span the errors occurred in.
- Click the Sample trace link to see the span details of an actual failed trace.
- Read the exception stack in the span's events section.
Following one particular request
- On the Distributed tracing screen, click Add filter.
- Choose Trace ID as the field, set the operator to =, and enter the trace ID as the value.
- Click the check button to apply the filter and the trace appears immediately on the Traces tab.
- Click the trace ID to see the request's whole path on the span waterfall chart.
Comparing performance across services
- On the Overview tab's table, compare each root service's request count, error rate and p50/p95/p99 response times.
- Read the relative performance differences between services visually from the response time bars beside the p99 column.
- Click the error rate badge of a service with a high error rate to go to its Error causes tab.
Related documents
- Applications — the distributed tracing tab of an individual application, showing only that application's traces
- Incidents — moving quickly to the traces involved in an incident
- Log viewer — connecting traces and logs to establish a cause
- Settings — setting response time SLO targets under inspection conditions