A.4. How correlation is measured
Pearson, Spearman, CCF, DTW, Granger
There are five ways of measuring "moved together". Each measures from a different angle.
The five at a glance
| Method | What it measures | Does it consider lag | Shape of the answer |
|---|---|---|---|
| Pearson | The strength of a straight-line relationship | No | -1 to 1 |
| Spearman | Whether the ranks move together | No | -1 to 1 |
| CCF | Correlation measured while shifting in time | Yes (a constant lag) | Best lag + correlation |
| DTW | The distance once stretched to match | Yes (a varying lag) | Distance (smaller is more alike) |
| Granger | Whether past values of one explain the other | Yes | p-value |
The default is Pearson.
1. Pearson — straight-line relationships
What it does
It measures how far two metrics move in proportion to each other.
| Value | Meaning |
|---|---|
| +0.7 to +1.0 | Strong positive correlation — they rise together |
| +0.3 to +0.7 | Weak positive correlation |
| -0.3 to +0.3 | Almost no relationship |
| -0.7 to -0.3 | Weak negative correlation |
| -1.0 to -0.7 | Strong negative correlation — they move in opposite directions |
A negative value is not meaningless. Some pairs are supposed to move in opposite directions, such as "free memory ↓ response time ↑". The sign is the direction; the strength is the absolute value.
A p-value comes with it
This is "the probability that the correlation is a coincidence".
| p-value | Meaning |
|---|---|
| < 0.05 | Less than a 5% chance of coincidence — taken as meaningful |
| ≥ 0.05 | Could be coincidence — treat with care |
A high coefficient with a large p-value means there were too few points, so it may have looked that way by chance. This is common when the analysis ran over a short window.
Limit — it misses anything that is not a straight line
There is clearly a relationship, but because it is not a straight line the Pearson value comes out low. This is where Spearman is needed.
2. Spearman — working from ranks
What it does
Instead of the values themselves, it converts them to ranks and then applies Pearson.
CPU values : 20 45 62 81 95
rank : 1 2 3 4 5
response time : 190 195 205 340 700
rank : 1 2 3 4 5 ← the ranks match perfectly
However uneven the spacing of the values, it only asks "when A was large, was B large too?" In the example above, Pearson comes out low while Spearman is close to 1.0.
Ties are handled with the average rank.
When it is useful
| Situation | |
|---|---|
| The relationship is curved | A metric that only reacts past a threshold |
| Badly departing values are mixed in | Converting to ranks reduces the effect of extremes |
| The units differ greatly | Ranks are compared with ranks, so it does not matter |
What the two values together tell you
| Pearson | Spearman | How to read it |
|---|---|---|
| High | High | A proportional relationship — simple and clear |
| Low | High | A curved relationship — there may be a threshold |
| High | Low | A few extreme values manufactured the correlation — be suspicious |
| Low | Low | No relationship |
The second and third rows are the useful ones.
3. CCF (cross-correlation) — shifting in time
What it does
It shifts one series one step at a time, measuring the correlation, and finds the point where it comes out highest.
Reading the result
| Best lag | Meaning |
|---|---|
| 0 | The two metrics moved at the same time |
| +2 | B followed A two steps behind |
| -3 | B moved three steps before A |
Why it is useful
Pairs that look uncorrelated when measured at the same instant come out clearly once shifted.
A pattern where response time rises tens of seconds after a GC run is not caught by a same-instant comparison. Shifting with CCF puts a number on that lag.
Whatever moved first is the more likely cause. This is the method used for tracing causes.
4. DTW — stretching the time axis to match
What it does
It stretches and compresses the two series to find the best overlap, then measures the difference that remains.
How it differs from CCF
| CCF | DTW | |
|---|---|---|
| Assumption | The lag is constant | The lag may differ from window to window |
| Answer | "shifted by n steps" | "this alike" |
| Value | A correlation coefficient | A distance (smaller is more alike) |
CCF shifts the whole series; DTW stretches and compresses window by window.
DTW is what catches situations where the propagation speed is not constant — something that happened quickly in an upstream service propagating slowly downstream.
Limiting the search
Stretch time without limit and any two curves start to look alike. A constraint called the Sakoe-Chiba band can be applied: "shift no further than this to match".
By default the whole space is searched without a constraint.
5. Granger — does it help the prediction
What it does
It tests "does knowing A predict B better?"
If ② is noticeably more accurate, then A contains information about B.
The name and the meaning differ
Despite being called a "causality test", it does not prove real causation.
All it can say is "A's past helps predict B." Two metrics both driven by a third factor can pass this test.
Why several methods are used together
Each method has its own blind spots.
| For a relationship like this | Method that misses it | Method that catches it |
|---|---|---|
| A curved relationship | Pearson | Spearman |
| A constant lag | Pearson · Spearman | CCF |
| A lag that varies between windows | Pearson · Spearman · CCF | DTW |
| Extreme values manufacturing a correlation | Pearson | Spearman |
| You want to know the order | Pearson · Spearman · DTW | CCF · Granger |
This is why MSA Failure Propagation Correlation uses all five. The rest compensate for the misjudgements you get from any one method (Chapter 402).
When the results disagree
| Disagreement | How to read it |
|---|---|
| Pearson low, Spearman high | There is a relationship, but it is not a straight line |
| Pearson high, p-value large | Too few points — widen the period and try again |
| CCF high but Pearson low | There is a lag — look at the lag value |
| Only Granger passes | It helps prediction, but the relationship is weak |
None of the methods proves causation
All five measure whether things moved together. None of them says that one caused the other.
Ice cream sales and swimming accidents both rise in summer. The correlation is high, but ice cream does not cause the accidents. The common cause is hot weather.
The same holds in operations. When traffic rises, CPU rises and response time rises with it. CPU and response time correlate strongly, but adding CPU will not fix response time if traffic stays the same.
Read the result as a narrowed list of candidate causes.
Further detail
The formulas for Pearson, CCF and Granger are in §5 of the algorithm guide
(600-algorithm_guide.md) in the Forecast MCP server documentation. Spearman and DTW are not in that document; the implementation (pkg/engine/correlator/spearman.go and dtw.go) is the reference.
Next
- Criteria for reading coefficients — Appendix: How to read the results
- Correlation scenarios — Correlation analysis
- How far it can be trusted — Appendix: Can you trust it