Too good to be true? Helping journalists assess scientific claims

New breakthroughs, radical treatment options, the costs and benefits of a new piece of technology: all of these events fall on the desks of daily news reporters in all markets and regardless of their beat or specialization. With that in mind, the American Medical Writers Association journal published guidelines in 1999 to help journalists who don’t have a background in science to better navigate a health science story.

Guyatt, G. et al. (1999) A journalist’s guide to writing health stories. AMWA Journal. 14(1):32-42.

Acknowledging the journalist’s instinctive caution, the paper moves the reporter beyond reflex decisions (i.e., peer-review = good / drug company release = bad) and outlines a series of questions journalists should ask anyone making a claim based in science. A summary of the questions appears below, but the article itself also explains what is meant by some of the terms.

Questions To Ask Investigators Making Claims About a Treatment

•Question about the study question

Is the alternative treatment the best available?

•Questions about the study methods

Were patients randomized to experimental and control groups?

Were patients in the two (or more) groups similar at the start of the study?

How many patients in the groups were “lost to follow-up”along the way?

Were patients and caregivers blinded to which group patients were in?

Were those measuring outcome blinded to which group patients were in?

Were those who conducted the statistical analysis blinded to which group patients were in?

•Questions about the results

How large is the effect of the exposure or treatment in reducing relative and absolute risk?

What were the side effects?

How much does the treatment cost?

•Questions about other studies

Have other studies addressed the question?

What were the results in those studies?

•Questions about conflict of interest

How was the study funded?

Has any party interested in this study’s outcome helped pay for your attendance at a scientific meeting?

Have you been a paid consultant for any relevant party?

Do you have a personal financial stake in the product you have studied