What does a “significant difference” between treatments mean?
Well, this is a trick question, because ‘significant difference’ can have several meanings.
First, it can mean a difference that is actually important to the patient. However, when the authors of research reports state that there is a ‘significant difference’ they are often referring to ‘statistical significance’. And ‘statistically significant differences’ are not necessarily ‘significant’ in the everyday sense of the word. A difference between treatments which is very unlikely to be due to chance – ‘a statistically significant difference’ – may have little or no practical importance.
Take the example of a systematic review of randomized trials comparing the experiences of tens of thousands of healthy men who took an aspirin a day with the experiences of tens of thousands of other healthy men who did not take aspirin. This review found a lower rate of heart attacks among the aspirin takers and the difference was ‘statistically significant’ – that is, it was unlikely to be explained by the play of chance.
What does “statistically significant” mean?
“To be honest, it’s a tricky idea..
On the basis of the evidence from the systematic review we can estimate that, if 1,000 men took an aspirin a day for ten years, five of them would avoid a heart attack during that time, but three of them would have a major haemorrhage.
Next: Obtaining large enough numbers in fair tests of treatments
GET-IT Jargon Buster
GET-IT provides plain language definitions of health research terms