9/03/14

The following article on evaluating treatment claims is from the Science-Based Medicine blog is being used with their permission. Since Discovery Eye Foundation provided the inspiration for Dr. Hall, we thought you might enjoy it as well.

I recently wrote about the claim that acupuncture can improve vision in patients with macular degeneration. In response, I received this e-mail:

“At Discovery Eye Foundation we have an education and outreach program for people with age-related macular degeneration, the Macular Degeneration Partnership. We are constantly getting calls from people who have heard of a new “cure” or a way to stop their vision loss. It is always hard to interject reality into the conversation and hear the hope leave their voice, replaced by frustration or despair.”

The e-mail suggested I write an article providing guidelines for consumers to help them evaluate the validity of treatment claims for themselves. On SBM we are constantly stressing the need to apply critical thinking to what you read, and the many pitfalls to be avoided. I’ll try to synthesize some of the principles into a handy list of questions.

What kind of evidence is there?

• If the claim is based on nothing but testimonials, STOP RIGHT THERE. You can forget about it, or at least file it away until there is better evidence. Anecdotes are not evidence; they only serve to suggest promising avenues of research. Science is the only reliable way to determine if a treatment is safe and effective.

• Is the claim based on a gold standard randomized, placebo-controlled, peer-reviewed study or some lesser kind of evidence like case reports? Was it published in a reputable mainstream medical journal? (If you’re not sure how reputable the journal is, you can look it up on tables of “journal impact factors.”

• Keep in mind that half of all studies are wrong. There are many factors that can lead to error. Preliminary or pilot studies that are positive are frequently followed by better, larger studies that are negative. We can never rely on one study without confirmation.

• Was it a meaningful clinical study in humans?

In vitrolab studies and animal studies may not be applicable to humans; if promising, they must be confirmed in good human studies. If it was a human study, did it show meaningful outcomes that made a real difference, like a reduction in heart attacks, or did it just show an improvement in lab values or risk factors?

• Were there 10 subjects or 300? Large studies are more trustworthy than small ones. The fewer dropouts, the better.

• Did it use an appropriate placebo control that subjects really couldn’t distinguish from the active treatment?

• Have other studies found similar results? Are there any studies that show the opposite? You can search PubMed and look for them.

Who is making the claim?

• Is it someone who is likely to be biased?
• Is it someone who provides the treatment or sells the product?
• Is it someone with expert medical knowledge or someone like the schoolteacher who claimed she had invented a cure for the common cold?
• Does the person have a good reputation, or a track record of making questionable statements?
• Is it someone who quotes or associates with unreliable sources like Mercola.com, the Weston Price Foundation, or the Health Ranger? Quackwatch has a useful list of non-recommended sources of health advice.

Where was the claim reported?

Real medical breakthroughs would be headline news. It’s not likely you would first hear about a cure for diabetes on an afternoon talk show or a Facebook page. Has your doctor heard of it? Are mainstream doctors recommending it? Is it covered on professional medical websites like the American Academy of Pediatrics or disease-focused websites like the American Diabetes Association? Is it sold only through multilevel marketing schemes?

What kind of language is being used?

Is it a sober factual report with caveats, or is it full of hype and buzzwords like “miracle,” “natural,” “known to the ancients,” “quantum,” “amazing,” “revolutionary”? If it really worked, advertising gimmicks wouldn’t be needed to sell it.

Does Gwyneth Paltrow swear it worked for her?

Testimonials can be very passionate but they are notoriously unreliable. Getting better when you use a treatment doesn’t necessarily mean you got better because of the treatment. Symptoms can fluctuate, diseases can resolve without treatment, and placebos can fool people. Perceptions can be wrong (think of optical illusions), the meaning of true perceptions can be misinterpreted, and memories can be inaccurate. Every snake oil salesman has reams of testimonials, and through the centuries there were testimonials galore for bloodletting to balance the humors. People frequently come to believe bogus remedies have worked for them. Barry Beyerstein wrote a classic article about that; it’s essential reading.

Does it make sense?

Is there a plausible mechanism of action? If someone claimed that standing on your head and whistling Dixie would cure diabetes, I think you would be skeptical. If it claims to work by a mechanism incompatible with known scientific principles, the level of evidence would have to be extraordinary for it to outweigh all the evidence those scientific principles are based on. Homeopathy’s claim that water can cure by remembering long-gone molecules, even after the water has been dripped onto a sugar pill and allowed to evaporate, would require extraordinary evidence indeed. If it’s a new antibiotic that is related to an old one, an ordinary level of evidence would suffice.

Is there a double standard?

Are they asking you to accept a “natural” or “alternative” treatment on the basis of the kind of evidence that you wouldn’t want the FDA to accept for allowing marketing of a prescription drug? There is only one science and only one standard of evidence.

Does it sound too good to be true?

Then it probably is too good to be true. Does it promise to cure a hitherto-incurable disease? Does it promise you can eat all you want and still lose weight? Does it promise there are no side effects of any kind? Does it remind you of the spiel of a used car salesman or a TV infomercial? Caveat emptor.

Who disagrees and why?

This is the most important question you can ask. It is rare for 100% of people to agree on anything. If you can find someone who disagrees, you can examine the reasons given for both opinions, and it will usually become obvious which side makes more sense. If you can’t find anyone who disagrees, it might be because it’s too new or because no scientist has taken it seriously enough to bother writing about it. In that case, withhold judgment and keep checking until someone does disagree.

Hope springs eternal, but true hope is better than false hope

If you are a desperate patient, it’s only natural to grasp at any straw of hope; but when the evidence is insufficient, the reasonable approach is to withhold judgment and wait for better evidence. You might think, “If it works, I don’t want to wait” but history teaches us that the great majority of these things don’t pan out. It might not do any harm, but then again it might; there might be adverse effects that haven’t been identified yet, it might raise false hopes only to dash them, and if nothing else it might waste time and money or interfere with getting more appropriate care. When you take an inadequately-tested medicine, you are essentially offering yourself as a guinea pig in a haphazard uncontrolled experiment that doesn’t even keep records. Of course, that’s your privilege; but I hope you would do it with your eyes open, with a realistic understanding of the state of the evidence.

Harriet A. Hall, MD
Retired US Air Force Physician
Editor of Science-Based Medicine Blog
Author of SkepDoc column in Skeptic Magazine