studies use flawed methodology

WASHINGTON (AFP) - More than one in three group-based cancer trials used a flaw statistical methodology to study the effects of an intervention, according to a study published Tuesday that reviewed 75 articles. only 75 studies number is probably higher than that

"We cannot say any specific studies are wrong. We can say that the analysis used in many of the papers suggests that some of them probably were overstating the significance of their findings," said David Murray, lead author of the review and chair of epidemiology in the College of Public Health at Ohio State University.Really ?? why not..chicken.. or your org will be lieable for all the faulty studies put out to change society..

But he emphasized that the studies did not willfully use inappropriate methods or try to skew results of a trial. Lier Lier what makes you think people are going to beleive that whooper

http://news.yahoo.com/s/afp/20080325/ts_afp/uscancerresearch_080325213104

Point of clarification:

Murray et al. reviewed 75 studies, of which 26 (35%) used inappropriate methods that would cause the authors to state an intervention made a difference when, in reality, it did not. The studies are not wrong (or faulty as you put it) in their whole methodology, but rather they miss a little on their analysis related to statistical significance.

What Murray et al. suggest in their paper is not that these studies are completely off and the magnitude of effect of the interventions are incorrect, but rather that there may be an overstatement of whether the intervention effect is meaningful.

As a personal point--from first glance most of the 75 studies appear to be about behavioral interventions (e.g., increasing physical activity among grade-school children and increasing colorectal cancer screening among members of church groups). I think that, regardless of whether the change is statistically significant, any sort of positive behavioral change in the area of cancer prevention is meaningful.

quote:

The studies are not wrong (or faulty as you put it) in their whole methodology, but rather they miss a little on their analysis related to statistical significance.

Okay.. that made me laugh out loud. I had a vision of reporting to funders or journals -- "Whoops, sorry guys, remember that statistically significant result I reported? Well... turns out I just missed a little in my analysis!"

LOL! My bad!

quote:

Originally posted by Russell:
As a personal point--from first glance most of the 75 studies appear to be about behavioral interventions (e.g., increasing physical activity among grade-school children and increasing colorectal cancer screening among members of church groups). I think that, regardless of whether the change is statistically significant, any sort of positive behavioral change in the area of cancer prevention is meaningful.

If it's not statistically significant, how can you say it's a positive behavioral change?

Or are you saying something else?

quote:

Originally posted by squeezer:
If it's not statistically significant, how can you say it's a positive behavioral change?

Or are you saying something else?

I'm saying something else. In regards to cancer prevention, particularly screening for colorectal, prostate, and breast cancers, any change towards the better would be positive in my opinion.

Statistical significance is based on sample size and strength of effect. Now, granted, strength of effect is a pretty big deal in studies (especially observational studies), but when dealing with intervention studies like these (which appear to be aimed towards decreasing cancer incidence and mortality among minority populations), I don't see why you should discredit the research just because the p-value isn't < 0.05.

From what I gathered from Murray et al. the group-randomization design seems to be relatively newer (i.e., it started gaining steam within the past 10 years), so there may be some bumps in the roads as people start to use the design and the design process gets ironed out. The design appears to use some pretty heavy analytical methods such as mixed modeling, so it doesn't surprise me that errors were made--in fact most of the errors reported in Murray et al. were in regards to correlation between members of the randomized groups during use of mixed modeling. While not accounting for this is incorrect, it's not a particularly egregious error, and it is one that can be easily fixed.

This is one of the upsides to peer-reviewed literature--your work is checked by others knowledgeable on the subject, allowing you to be made aware of your errors and, if you are able, fix those errors you have previously made. I'm interested to see if any of the authors from the reviewed studies respond to Murray et al.

Props to Jemeyes, though. The article was very interesting, and it is an issue in peer-reviewed literature that needs to be addressed more--especially in terms of clinical trials. I wonder if similar review articles have been published for clinical trial studies.

Any competent statistical analyst knows to examine and "take into account any similarities among group members or any common influences affecting the members of the same group" (from the Science Daily press release). That's not exactly rocket science. Nor is group randomization design new.

However, the pressure to produce statistically significant results influences choices in analysis. That's the actual problem. Whether Murray says it out loud or not.

quote:

Originally posted by Ariel:
However, the pressure to produce statistically significant results influences choices in analysis. That's the actual problem. Whether Murray says it out loud or not.

Your point is dead on, Ariel. Studies with non-statistically significant results aren't published as often than those with significant results. This seems especially true for the bigger journals such as NEJM, JAMA, and Science.

And I think the problem Murray exposes is that the analyses are being performed by people who may not exactly be experts in statistics--I'm sure it happens all the time and seems to be one of the major points by the study. In the future it would help the research field to include on their research team a methodoligst or statistician who knows how to validly perform the analysis.

Russell, I was for many years (okay, I guess I still am p/t) a statistical analyst. And even those of us who are experts are just as susceptible to pressure. (We all work for a living, ya know...?) In addition, one doesn't need to understand even the programming anymore, which really means that stats can be run by folks who don't even understand the mathematics (or theory). They just know how to push buttons.

I agree there is nothing Holy about .05, or .01 or .001. Although, as so often multiple statistical test are often performed with no adjustment, you could also argue that these levels should be tightened.

The problems with public health research (from incompetence to outright fraud and corruption) are so broad that I wouldn't even know how to begin to describe them.

Also, obviously, I've never been involved with a study that didn't employ a stats/methods expert... since that would have been me. But in my time with gov't and universities, I can't think of one I've heard of either. So, I assume there is usually someone who at least checks the stats.