15 to 42 percent of medical research are false positives (Yet Another Calculation)

A while ago I found a very interesting paper from Leah R. Jager and Jeffrey T. Leek  via a post in the Simply Statistics blog arguing that most published medical research is true with a rate of false positives among reported results of 14% ± 1%.  Their paper came as a response to an essay from John P. A. Ioannidis and several others authors claiming that most published research findings are false.

After dealing with some criticisms Mr. Leek made a good point in his post:

“I also hope that by introducing a new estimator of the science-wise fdr we inspire more methodological development and that philosophical criticisms won’t prevent people from looking at the data in new ways.”

And thus, following this advice, I didn’t let criticisms prevent me from looking at the data in a new way. So for this problem I have devised a probability distribution for p-values to then fit the data via MLE and infer from there the rate of false positives.

pvalues PDF CDFSo this is my take; 15.33% rate of false positive with a worse case scenario of 41.75% depending on how mischievous researchers are but, in any case, and contrary to what others authors claim, most medical research seems to be true.

Continue reading

Climategate (2/3): be careful what you model for because you might get it

In the previous post I showed how James Hansen at GISS NASA clearly over estimated global warming in the late 80’s due to the modeling choices he made. To make a point on how influential the choice of a model is, in this post I will make modeling choices that will allow us to claim that global warming can be explained as a fluke in a random process.

I like to explain the relationship between data and models saying that data is the shadow reality casts, and models are what we believe is casting the shadow. So once we have a model  we can use it to cast shadows (make predictions) like the one James Hansen did and could be read in 1986 newspapers:

Hansen predicted global temperatures should be nearly 2 degrees higher in 20 year. “Which is about the warmest the earth has been in the las 100,000 years.”

Interestingly James Hansen downgraded his prediction in a 1988 paper from the nearly two degrees higher to a one degree higher. Though to be fair I would not be surprised if media misquoted him; I might not trust scientists but I absolutely distrust media.

Anyhow, let’s now compare NASA’s prediction in this 1988 paper (in red) to what actually happened years later (in blue):

Yearly Average Global Temperature Change
1988 NASA predictions on top of yearly average global temperature changes with major volcano activity and parts per million levels of CO2

Data for this plot comes from the B.E.S.T and N.O.A.A.

Continue reading

Climategate (1/3): be careful what you model for because you might get it

Deception is all around us, in every little parcel of our life; from our personal Bart Simpson’s “It wasn’t me” to our local TV news host selling us the latest “You’re not going to believe this” but we eventually do. One might just wish there would exist communities out there with higher standards like, for example, Christians priests but, nope, they cover up pedophile networks in order to preserve The Church’s “good” name. But how about the atheist priests a.k.a scientists? How about their standards?

Well, unfortunately the community of scientists might have more to do with priesthood than one might expect or desire, and a nice example of this would be the Climategate (or the Climatic Research Unit email controversy, as some people had the kindness to rename the Climategate article in the Wikipedia following Fox News’ motto “fair and balanced” )

So in this post I am going to replicate earlier studies on global warming to uncover how over pessimistic were the maths models of the past, but I will also talk about human weakness, and scientists are human… for now.

Mike's Trick
Click to watch Dr. Richard A. Muller (Professor of Physics at the University of California at Berkeley) talk about “Mike’s Trick” and the Climategate.

Continue reading