I was reading about some reasons to distrust meta-analysis, and when I went to evaluate the ideas' merits, I remembered a source of error that I'd read about recently. I enjoy Eades' epistemological takedowns, but this one basically amounts to, "Meta-analyses are usually done wrong."
Eades is correct to dismiss meta-analyses, though perhaps not for the reasons he thinks. (I suspect experience guided him.) Although he has good reasons; a meta-study including methodologically poor studies is going to be a poor meta-study, and a meta-study has to be very careful to include data fairly. If it fails to do either, I would call it a mistake, not a study; it's quite possible for a meta-analysis to do both properly.
This mainly impacts journalistic writing, as they'll happily write on any study that fits their agenda, and generally forget caveats and other subtleties. If you're in a position to read the actual paper, you're in a position to tell if they're cherry-picking data.
However, how many meta-studies do you suppose correct for the fact that neutral or negative studies don't result in papers? I'm not sure how I would check, but I'm going with negative zero. They can't even try to deal with cherry-picking that occurred before the paper is published.
It's a shame. Yesterday, the meta-study was my favourite kind of statistical study.
No comments:
Post a Comment
New failcomment system also fails to publish my comments, it's not limited to yours. Keep trying, it will usually work, eventually.
Blogger deliberately trying to kill itself, I expect.
Captchas should be off. If it gives you one anyway, it's against my explicit instructions.