Skip to content

A.5. Can you trust it

MAPE, RMSE, R², and the conditions that change the result

The measures of how well it fitted​

How closely a forecast matched reality is measured five ways. These sometimes appear in the answer.

The same forecast measured five ways​

        actual   forecast   error
          100        95        5
          200       210       10
          150       148        2
MeasureCalculationResultMeaning
MSE(5² + 10² + 2²) ÷ 343Squared units, so not intuitive
RMSE√436.56"wrong by about 6.56 on average"
MAE(5 + 10 + 2) ÷ 35.67"the average size of the error"
MAPE(5% + 5% + 1.3%) ÷ 33.8%"wrong by 3.8% on average"

RMSE is larger than MAE. Squaring penalises large errors more heavily. A wide gap between the two is a sign that it was badly wrong a few times.

R² — how much is explained​

R² = 0.95   explains 95% of the variation in the data
R² = 0.50   explains half
R² = 0.10   barely helps
R²Assessment
0.9 and aboveVery good
0.7 to 0.9Good
0.5 to 0.7Middling
Below 0.5Needs improvement — treat the result as indicative only

Which one to look at​

What you want to knowMeasure
Comparing against another metricMAPE — being a percentage, it compares across different units
Avoiding large mistakesRMSE — large errors are penalised heavily
The average size of the errorMAE
Whether the trend is explainedR²

MAPE grows sharply when the actual value is close to zero, because the divisor shrinks. A MAPE of several hundred percent in the early hours, when there is almost no traffic, is not surprising. Look at RMSE in that case.

Six conditions that change the result​

If any of the following applies, treat the result as indicative only.

① The evidence window is short​

A time range of one hour means the calculation used a few dozen points. Forecasting is particularly weak over short windows.

→ Widen the range and ask again. Three days is the maximum.

② There was a deployment or an event in that window​

Using a window where the trend was broken throws the projection off. In anomaly detection, such windows are flagged even though they are normal.

→ Ask again over a window that avoids it.

③ There is a gap in collection​

Missing values make the calculation inaccurate. Check whether the chart has an empty stretch.

④ The value barely changes​

If it is always the same, there is no basis for separating "normal" from "different." Correlation also fails to compute, or produces a meaningless value.

⑤ The metric combination does not fit​

Custom Setup runs exactly what you chose. Combining two unrelated metrics still produces numbers (Chapter 305).

⑥ The wrong target was picked up​

You may have entered the wrong value in the confirmation dialog, or the same name may exist in several places. Check the target name written in the answer first.

How much to trust what​

ResultDegree of trust
Queried values (metrics, table structures, raw logs)Actually read from the source — high
Calculated results (correlation coefficients, forecast lines, anomalous windows)Depends on the evidence window — check the six above
Interpretation and suggested actionsIndicative — always verify

The further down, the more checking is needed. Read the evidence behind a number, and review any command before you run it.

Three habits for checking​

  1. Expand CogentAI's thoughts above the answer
  2. Check that the charts and tables match the answer
  3. If it looks wrong, change the period and ask again — the same answer twice raises confidence

And the notice that is always shown​

The notice always shown on screen is not a formality (Chapter 101). Read it once before running any command as given.

Next​