5.1. Incidents
The alerts raised automatically when an SLO (service level objective) is breached or a system anomaly is detected, and the analysis of their root cause.
Overview
The Incidents menu shows every incident OPENMARU Observability detected automatically. An incident is an alert raised when a service level objective (SLO) is breached or an application anomaly is detected.
On this screen you can look at both ongoing and resolved incidents and check each one's severity, duration, the proportion of requests affected and how much of the error budget it consumed. Click an incident to open its detail page, where root cause analysis (RCA) helps you establish why it happened.
Screen layout
The Incidents page is made up of the following areas.
- Top header: the application filters
- Status summary strip: incident counts by severity, and the option to show resolved incidents
- Incident list table: every incident, with its SLO impact
Main features
Reading the status summary strip
The status summary strip at the top of the screen gives the incident count per severity at a glance.
| State | Meaning |
|---|---|
| Critical | Incidents needing immediate action |
| Warning | Incidents needing attention |
| Resolved (critical) | Incidents resolved from a critical state |
| Resolved (warning) | Incidents resolved from a warning state |
Click an entry to filter the list to incidents in that state. Click again to clear the filter.
Tip: clicking Critical gets you quickly to the incidents that need an immediate response.
Showing resolved incidents
By default only ongoing (unresolved) incidents are shown. Tick Show resolved incidents to include resolved ones as well.
Resolved incidents appear translucent in the list, with a Resolved badge in the duration column.
Note: the show-resolved setting is saved in the browser, so your last choice persists into your next visit.
Old incidents are removed automatically
Incident and alert history is deleted by the server once it is older than the metric cache retention period plus two days. With the default settings (a 30-day cache) that is about 32 days.
The cut-off follows the cache because an incident is of little use once the charts behind it are gone. The history and the evidence disappear together.
Note: unresolved incidents and alerts are never removed, however old they are. Anything that has dropped off the list was resolved.
To keep history longer, raise the server's cache retention (cache-ttl). That needs proportionally more
metric storage.
Application filters
Use the application filters at the top right of the page to show incidents for particular applications or namespaces only. You can also search by incident ID or keyword to find one quickly.
The incident list table
The incident list table carries the following.
| Column | Description |
|---|---|
| Incident | The incident ID (for example i-123), coloured by severity. |
| Application | The application the incident occurred on |
| Namespace | The namespace the application belongs to |
| Kind | The Kubernetes workload kind (Deployment, StatefulSet and so on), or ExternalService |
| Opened | When the incident was first detected, and how long ago |
| Duration | How long the incident lasted. Ongoing incidents get an Unresolved badge, finished ones a Resolved badge |
| Availability | The availability SLO compliance rate. A breach is highlighted in red. |
| Response time | The latency SLO compliance rate. A breach is highlighted in red. |
| Requests affected | The proportion of requests the incident affected (as a bar) |
| Error budget consumed | How much of the error budget was consumed (as a bar). Above 100% it turns red. |
Note: the availability, response time, requests affected and error budget consumed values come from further analysis of the incident data, so a spinner may appear while they load.
Click a column header to sort by it, ascending or descending.
Adjusting SLOs
Click the more (...) button at the right of an incident row to adjust that application's SLO thresholds quickly.
- Adjust availability SLO: change the availability SLO threshold
- Adjust response time SLO: change the latency SLO threshold
- Resend alert: resends the alert (shown for unresolved incidents only).
Note: an SLO threshold adjustment applies to that application only. Change the defaults for everything under Settings > Inspection conditions.
Incident details
Click an incident ID or an application name in the list to open that incident's detail page.
Header information
The top of the detail page carries the following.
- Incident ID: a unique identifier prefixed with
i- - Severity badge: critical or warning
- State badge: Still ongoing while it continues, Resolved once it has finished
- Metadata: application name, namespace, start time, duration
Click the Incidents link to return to the incident list page.
Incident details
The Incident details section lays out the incident's basic properties as a grid.
| Item | Description |
|---|---|
| Severity | Critical or warning, as a coloured badge. |
| Application | The application affected. Click it to open that application's detail dialog. |
| Started | When the incident was first detected, and how long ago |
| Resolved | The resolution time, or the Still ongoing state |
| Duration | The total duration of the incident |
| Category | The application category. Click it to open that application's detail page. |
Service level objectives (SLO) section
The Service level objectives (SLO) section shows which SLO the incident occurred on, as a table.
| Column | Description |
|---|---|
| SLO | The SLO's name (availability or response time). It carries a green check or a red warning icon depending on whether it was breached. |
| Compliance | The actual SLO compliance rate. A breach is highlighted in red. |
| Target | The SLO target (for example "99% of requests must be served within 500 ms"). Click the pencil icon to edit the SLO threshold directly. |
Analysis tabs
Two analysis tabs sit at the bottom of the incident detail page.
| Tab | Description |
|---|---|
| Root cause analysis (RCA) | The system's automatic analysis of the incident's cause. Selected by default. |
| Traces | Trace data from the period the incident covered |
The RCA (root cause analysis) tab
The RCA tab is selected by default on the incident detail page and shows the system's automatic analysis of why the incident happened. It has five sections; four of them — SLI, propagation path, causal timeline and detailed report — can be collapsed and expanded, the root cause summary excepted.
1. Service level indicators (SLI) section
The service level indicator (SLI) section charts the state of the service over the incident's analysis period. The section header carries the target application's name and the incident ID.
It contains two sub-areas.
The response time heatmap
Visualises the distribution of response times over the incident period as a heatmap. The X axis is time, the Y axis is response time, and colour intensity is the density of requests in that cell. Drag to select a region on the heatmap and the application detail dialog opens, where you can look at the trace data for that period.
The SLI charts
A response time chart and an error rate chart sit side by side, showing how the service's performance moved over the analysis period.
- Response time chart: how request response times trended over the analysis period
- Error rate chart: how errors trended over the analysis period
A note below the charts explains that they cover the analysis period's SLIs.
Tip: drag to select a period on the SLI charts and you get a detailed analysis of it (an RCA confined to that time range). The selected period is highlighted on the chart.
2. Root cause analysis summary
A card summarising the system's automatic analysis of the incident's cause. It has three parts.
Findings by severity
Badges at the top right of the card give the number of findings by severity.
| Badge | Meaning |
|---|---|
| Critical | Findings needing immediate action |
| Warning | Findings needing attention |
| N found | The total number of findings |
The estimated root cause
The most likely cause of the incident is shown in a highlighted card, containing the following.
- Cause title: what is believed to be the root cause (a deployment change, resource saturation, an upstream failure, and so on)
- Application name: the application the root cause occurred on
- Description: further explanation of the cause (the deployment at a particular time, an error count, and so on)
Categorisation
The findings are grouped into the following categories and shown as chips, each carrying that category's finding count.
| Category | Description |
|---|---|
| Deployment | Causes relating to deployment changes (a new version, a rollout event) |
| Upstream | An upstream service's failure or SLO breach affecting this service |
| Resource | Causes relating to resource saturation — CPU, memory, disk |
| Log | Causes relating to anomalous log patterns |
| SLO | Another service's SLO breach affecting this one in a chain |
| Database | Database-related issues |
3. Issue propagation path
Where an incident affected several applications, this visualises the path the failure propagated along as a service dependency map. The section header gives the number of applications involved.
Reading the propagation path
- Nodes (applications): each rectangle is one application. The node's border colour is that application's state (red: critical, yellow: warning, green: healthy).
- Arrows (connections): the direction of traffic between applications. The arrow's colour is that connection's state. Connections in a critical or warning state carry a flow animation.
- Incident marker: the application the incident occurred on (the target application) carries a crosshair icon.
- Root cause marker: the application believed to be the root cause carries a star icon.
- The causal path: the nodes and connections on the causal path from the root cause to the incident target are highlighted.
- Traffic statistics: traffic statistics labels sit on the connections. Hover over an application and only the services directly connected to it stay highlighted; the rest dim.
Tip: click an application name to open its detail dialog and analyse its metrics, logs and distributed traces further.
Note: you can zoom the map area with the mouse wheel and pan it by dragging. Double-click an empty area to restore the default position.
4. Causal timeline
Lists the related events either side of the incident in time order, so you can follow the chain of causation that led to it. The section header gives the event count.
Switching view mode
The causal timeline offers two view modes, switched with the buttons at the right of the section header.
Compact view
Summary statistics for the events appear at the top.
| Statistic | Meaning |
|---|---|
| Events | The total number of events on the timeline |
| Critical | Events at critical level |
| Warning | Events at warning level |
| Estimated root cause | Marks the event believed to be the root cause |
Below the statistics, each event is listed compactly on one line carrying the time, a severity dot, the application name and the event title. The root cause event gets an Estimated root cause badge, and the moment of the incident gets an Incident badge.
Full view
Each event is shown in detail as a card, arranged vertically along the time axis. Each event card carries the following.
- Time: when the event was detected
- Severity marker: a coloured round marker for the severity (red: critical, yellow: warning, green: informational)
- Category chip: the event's cause category (deployment, database, resource, upstream, log, SLO)
- Severity chip: the event's severity level
- Application link: the related application's name (click to open its detail dialog)
- Event title: the finding's title (for example "SLO: availability", "Deployment change: v2.1.0")
- Description: further information about the event
The card for the event believed to be the root cause is highlighted with a red left border and a blinking effect, and carries an Estimated root cause badge.
The moment of the incident is inserted into the timeline as its own incident card, so you can see visually which events came before and after. Where the incident is still running, In progress appears instead of an end time.
Tip: comparing the time of the root cause event with the time of the incident on the causal timeline tells you the interval between cause and effect.
5. The detailed RCA report
Presents the analysis of each individual finding as a tree, letting you explore the incident's causes hierarchically.
The incident time range
A bar at the top of the detailed RCA report gives the incident's time range. It carries the target application's name and the incident's start and end times; where the incident is still running, In progress appears instead of an end time.
Understanding the tree
The tree's root node is the SLO breach on the application the incident occurred on. Below it sit group nodes for each cause category, and below each group, the individual finding nodes.
What a tree node carries
Each node shows the following.
| Element | Description |
|---|---|
| Collapse/expand arrow | Where the node has children, click to collapse or expand them |
| Node name | The category name or the finding's title |
| Application link | An icon that takes you to the related application (click to open its detail dialog) |
| Estimated root cause badge | A red badge on the node judged to be the root cause |
| Incident badge | Shown on the incident's target node |
| Possible cause icon | A red warning icon on nodes that may be a cause |
| Confidence badge | The confidence level of the cause (high, medium, low) |
| Sparkline | The item's time-series data as a small line chart. The incident's period is shaded red. |
| Time range | The period over which the event occurred |
Category group nodes
The category group nodes gather the findings by type.
| Category | Description |
|---|---|
| Deployment changes | Deployments and rollout events during the analysis period |
| Upstream service issues | A failure or SLO breach in an upstream service |
| Resource saturation | Saturation of infrastructure resources — CPU, memory, disk |
| Log anomalies | A spike in error or warning level log patterns |
| SLO chain | Another service's SLO breach affecting this one in a chain |
| Database issues | Degraded database response times, connection problems and so on |
Finding nodes
The leaf nodes, the findings, carry the specifics of each cause.
- Application name: the application the cause occurred on
- Report category: the relevant report area (SLO, CPU, memory, network and so on)
- Inspection item: the name of the inspection condition whose threshold was passed
- Sparkline: how that metric trended. Hover over the chart and a tooltip gives the value and time at that point.
Each finding's sparkline also shades the incident's period in red with a dotted line, so you can see visually how the cause event and the incident relate in time.
Looking at a node in detail
Click a finding node and a detail dialog opens. What it offers depends on the type of cause.
Log pattern details
Click a finding in the log anomaly category and a log pattern dialog opens carrying the following.
- Severity: the log level (critical, error, warning, info, debug)
- Occurrences: how many times that log pattern occurred in total
- Time-series chart: how the log pattern's occurrences trended
- Sample messages: actual log messages matching that pattern
Click View messages at the bottom of the dialog to see every log message with the same pattern on that application's logs tab.
Metric details
Click a metric-based finding — a resource, an SLO — and that metric's detail widget (a chart) appears in a dialog, so you can compare visually how the metric moved either side of the incident.
The traces (distributed tracing) tab
The Traces tab of the incident details shows trace data from the period the incident covered. The trace heatmap and trace list are filtered automatically to the incident's application and its time range.
Where a response time SLO is configured, traces that passed that threshold are filtered in automatically. Click a trace to see its span details.
Tip: analysing the traces whose response time spiked lets you verify the root cause RCA estimated, at the level of an actual request flow.
The incident investigation workflow
To investigate an incident effectively, follow these steps.
- Read the situation from the incident list: check the critical and warning incident counts on the status summary strip.
- Choose the incident: use the severity or application filters to pick the incident to investigate.
- Check the SLO breach: on the incident detail page, check the compliance rates of the availability and latency SLOs.
- Establish the cause from the RCA summary: read the estimated root cause and the categorisation in the root cause analysis summary on the RCA tab.
- Analyse the propagation path: see from the issue propagation path where the failure started and where it spread.
- Review the causal timeline: work out from the causal timeline how the root cause event and the incident relate in time.
- Go deeper: where necessary, explore the individual causes as a tree in the detailed RCA report, or analyse the actual request flow on the distributed tracing tab.
- Move to the application details: click an application link to analyse its metrics, logs and distributed traces further.
The Alerts menu
The Alerts menu sits immediately below Incidents in the left sidebar. Where an incident is the higher-level event raised automatically by an SLO breach or anomaly detection, an alert is the notification raised when an individual inspection item or a user-defined alerting rule fires. On the Alerts screen you can look at the alerts that fired, resolve or pause them, and manage the rules that raise them.
The screen is split by the tabs at the top into Alerts and Alerting rules.
The Alerts tab
Shows firing and resolved alerts as a table.
- Status summary counters: firing counts by severity (critical, warning) at the top left; click one to filter to that severity.
- Show resolved alerts: only firing alerts are shown by default; turn the toggle on to include resolved ones.
- Security alerts only: filters to alerts raised by security attack detection.
- Application filters: use the category and namespace filters at the top right to look at one group of applications only.
The list table has the following columns.
| Column | Description |
|---|---|
| Alert message | The alert summary and its ID (a-...). The icon before it gives the state — firing (mdi-bell-alert), resolved (✓ mdi-check-circle), paused (mdi-bell-off) |
| Application | The application the alert fired on. Click it to open the application details |
| Namespace | The namespace the application belongs to |
| Kind | The workload kind (Deployment, DaemonSet, StatefulSet and so on) |
| Edit rule | The edit icon for the rule that raised this alert. Click it to go to the rule editor |
| Fired at | When the alert first fired, and how long ago |
| Duration | How long it has been firing, and its current state (firing / resolved) |
| Severity | WARNING or CRITICAL |
Select several alerts with the checkboxes and a bulk action bar appears at the top, letting you resolve, pause or reopen them all at once.
Alert details
Click an alert row to open its detail dialog.
- Header: the severity badge and the current state (firing / resolved / paused)
- Tags: alert ID, application, namespace, workload kind, alerting rule name
- Alert message: a human-readable summary of the firing condition
- Where it fired: the application the alert is attached to (click the link for the application details)
- Timing: fired at, duration, data source (for example
Logs,SLO) - Action buttons: Resolve (resolve manually), Pause (suppress the alert for a set time), Resend alert (resend a currently firing alert to external channels such as Slack and Teams, and as an on-screen toast)
The Alerting rules tab
Click Alerting rules in the tabs at the top to see the rules that raise alerts.
- The count of enabled and disabled rules appears at the top; the ＋ button at the top right adds a new rule, and the export icon exports the rules as JSON.
- The icon before a rule name distinguishes three kinds.
- Lock (
mdi-lock) — a config-managed rule, deployed through a configuration file and not editable or deletable from the screen. - Shield (
mdi-shield-check) — a built-in rule, created automatically by the system at first installation. It can be edited and deleted just like a user-defined rule (the difference being that its name is shown translated). - Bell (
mdi-bell-ring) — a user-defined rule, fully editable and deletable.
- Lock (
| Column | Description |
|---|---|
| Rule name | The rule's name (shown in Slack and Teams messages). Built-in rules are shown translated |
| Data source | The kind of source the rule evaluates (inspection item, log pattern, PromQL, Kubernetes event and so on) |
| Severity | Warning or critical |
| Selector | What the rule applies to (all applications / by category / individually chosen) |
| Currently firing | How many alerts are firing from this rule (click to filter to them) |
| State | The rule's enabled/disabled toggle |
At first installation the system creates about forty built-in rules automatically, covering SLOs, CPU, memory, storage, network, instances, databases, runtimes, DNS and logs. Built-in rule thresholds can be overridden at project level under Settings > Inspection conditions.
Configuring an alerting rule
Click the ＋ button (or edit a user-defined rule) to open the rule configuration dialog.
| Setting | Description |
|---|---|
| Rule name | The identifying name shown in alert messages (for example Order service response delay) |
| Data source | The source the rule evaluates — inspection item (the state of a built-in inspection), log pattern (aggregated errors and warnings in logs), PromQL (a metric query expression), Kubernetes event (aggregated events such as FailedScheduling), attacking IP (security detection), resource forecast (predicted resource exhaustion) |
| Severity | Warning (needs attention) or critical (needs an immediate response) |
| Application selection | All applications / a particular category / individual apps (glob patterns) |
| Timing | The pending period and the resolve grace period (see below) |
| Enabled | Whether the rule is active |
Timing: the pending period and the resolve grace period
Two timing settings that prevent unnecessary alerts (alert fatigue) and flapping.
- Pending period (For): how long the condition must hold continuously before an alert fires. If the condition clears while pending, the wait is cancelled. This filters out false alerts from momentary spikes. Set it to
0and the alert fires the instant the condition is detected. - Resolve grace period (KeepFiringFor): how long the alert is held open after the condition clears. If the condition is met again during the grace period, the alert stays open. This prevents the flapping you get when a condition clears and returns repeatedly.
| Pending | Grace | Behaviour |
|---|---|---|
| 0 s | 0 s | Fires immediately, resolves immediately (sensitive) |
| 300 s | 0 s | Fires after 5 minutes of the condition, resolves as soon as it clears |
| 0 s | 300 s | Fires immediately, stays open for 5 minutes after clearing |
| 300 s | 300 s | Fires after 5 minutes of the condition, stays open for 5 minutes after clearing (stable) |
Connecting alert channels
To send firing alerts to external channels such as Slack, MS Teams and webhooks, configure the channel under Settings > Alert channels. For each channel you can choose whether it receives incidents, deployments and alerts, and the alert language (Korean or English). For details, see Settings — alert channels.
Related documents
- Applications — per-application metrics and SLO status
- Settings — inspection conditions — setting SLO thresholds and inspection conditions
- Settings — alert channels — configuring Slack, MS Teams and webhook alert channels
- Timeline map — incidents and deployment events in time order
- Distributed tracing — detecting response time anomalies with the trace heatmap
- Chart guide — how to read charts and the shared controls