Four Tails and You’re Out
ANOTHER hospital has called in the Healthcare Commission because it has suffered a run of deaths in its heart transplant programme. This time it was Harefield in North-West London, where four consecutive patients died between 17 July and 3 September 2008.
Hospital managements tend to panic when this happens, but it usually means nothing. There is no statistical law that says deaths should be spread out evenly across the year: bunching is quite normal. Try tossing a coin 100 times and see how many runs of heads or tails appear. How many runs of four would you expect in 100 tosses? The answer is far more than most people expect.
In any case, for death to be a worthwhile marker, there need to be far more operations every year than any heart transplant unit carries out. To detect even a doubling of the death rate and have confidence that the answer is statistically significant needs at least 200 operations a year. Harefield did 15 in the whole of 2008. The whole of the UK did only 288 in the 32 months covered by the review.
Trying to extract meaning from a run of four deaths using statistical methods is a mug’s game. The Healthcare Commission report cites the UK Cardiothoracic Transplant Audit, which carries out a running check on outcomes, using a technique called tabular CUSUM methodology. Call it what you like, it isn’t going to generate hard evidence from a handful of cases. Pretending it is does a disservice to surgeons, and to statisticians.
The report more or less admits this, saying: “Differences and changes in mortality which may appear large are not necessarily identified as significant from a statistical perspective.”
So the run of deaths tells us nothing about the quality of the surgeons or the operations they carried out. Calling in an inquiry team may reassure patients and public – and it’s right that surgeons should be concerned about their standards – but it isn’t evidence-based medicine. It demonstrates ignorance of the laws of probability and statistical significance, wastes time and money, causes anxiety and adds nothing much to the quality of care. But it’s happened before – at Papworth Hospital, and in Scotland - and it’ll happen again
Simon Coward (not verified) wrote,
Thu, 18/06/2009 - 09:39
I’m afraid I cannot let the fallacies in this piece pass without comment, they are at least as problematic as those the author sees in the original report.
For the comparison with a coin-toss to be valid, you would not only need there to be just two outcomes, you would need both of them to be equally likely. While at this comparatively superficial level there are only two outcomes – the patient either survives or does not – the average likelihood of surviving for at least a year after the operation appears to be nearer 90% than 50%. As a result, with a fatality rate of nearly 27%, four deaths out of fifteen operations in 2008 would appear in itself to be an aberration worth looking into. With those probabilities, four consecutive deaths would still be quite unlikely if you performed a hundred operations, out of fifteen I would have thought they might be an indication of something quite catastrophic and certainly, as human lives are at stake, well worth investigating.
David Harris (not verified) wrote,
Thu, 18/06/2009 - 17:15
Also there is a problem with ignoring any adverse outcome until it is significantly beyond doubt. We continuously analyse our outcomes in our cardiac unit to try to pick up a POSSIBLE bad trend. If we see one, we don't panic but we do raise our eyebrows. If it then gets better, fine (as it always has done so far), if it continues then we escalate our level of investigation.
As deaths actually on the operating table are now so rare, we investigate each one (just as the airlines do), but again we are cautious in what inferences we draw.
One of the problems of both Bristol Childrens' and Dr. Shipman is that people did ignore the early (and totally insignificant statistically) warnings.
Would you be happy to fly with an airline which ignored a plane whose wings fell off? With an N of 1, almost no inference can be drawn, but I would prefer them to check their other planes of the same type!!!
Post new comment