Thursday, April 19, 2018

"Replication" Is Not a Malleable Standard

As I start reading over these Genome-wide Association Studies, I am seeing a lot of attempts to "replicate" their findings within their study.  This is something I assume is being done in order to dispense with the annoying problem of putting out studies that are never replicated.  "How do they do this"?, you might ask.   And this is quite interesting.  They use meta-analysis.  They take old studies that actually had negative results, combine them and compare the meta-analysis results to the alleged positive findings.  (Addendum:  I now will refer to these attempts at replication as "Hindsight Replication").  "Do they have to match the findings of the study, with statistically significant results"?, you might ask. 
Well, no, apparently they only need to be "trending in the same direction" (the data, while not necessarily significant, are leaning the same way).  "Do all of the results need to be trending in the same direction"?, you might ask.  Well, no, just an undefined percentage of them.
I will agree that a GWAS, even if you accept their validity, would never be exactly duplicated.  That is why GWAS's should only serve as screening tests.  If you find genes that appear to have a mathematical significance, then you should then try to replicate that by doing a study on a NEW dataset for just that one gene.  Of course, even that isn't a perfect replication, but that is where you should start.  If you actually get a positive result, I think you can at least fairly claim replication at that point, although I think you should do it one more time for the books, at least.
This technique is not "replication" and should be called something else.  You can't take studies that failed to find the results you currently found and use them to claim you replicated anything.
Here's a good way to realize that you are simply manipulating data.  Ask yourself why the people doing the study don't do it the other way?  In other words, find out which new gene linkages you come up with in the meta-analysis and then see if you get the same with your new dataset.
Deep down, you know that is a more honest attempt to replicate.  Otherwise, these are just data masturbations.
Anytime you deliberately lower your standards for replication, you are trying to get the study to do what you want it to do. 

No comments:

Post a Comment