Scientific Method

Published on October 21st, 2013 | by Rayne

2

Researching the research – How to divorce pseudoscience from science and separate fact from fiction

Sorry I’ve been away for so long, dear readers. I’ve been busy with life and finding a new job.

I thought I would post my submission to the Atheist Foundation of Australia magazine. Feedback is welcome!

Researching the research – How to divorce pseudoscience from science and separate fact from fiction.

Ten years ago, my dial up modem was my gate way to what I thought was fast way to access information from around the world. Ten years on, internet technology has seen a vast amount of growth – providing faster access to an even greater expanse of information. I foolishly thought that this would be the start of a revolution of critical thinking. No longer would I have to explain to family members how flu shots worked. I thought wrong.

In an age where I can access information in any language from my smartphone while eating a bagel on a train, I’m still subjected to posts on Facebook from friends asking which naturopathic remedies they could take for blue lips and severe stomach pain. Social media it seems has made it easier to spread information – just not accurate information.

Pseudoscience has successfully managed to infiltrate the lives of many. Often dressed up in a darling sheep’s costume, the convincing wolf that is pseudoscience has slandered its way into the medical field – mingling with real science and tricking the world into believing that paying $60 for a bottle of shampoo that claims to be formulated from “unique amino acids and all natural placenta” will make your hair look like Scarlett Johansson and smell like a bowl of petunias.

Unfortunately since science has never promoted itself as having all the answers – which is true, science doesn’t have all the answers., this allows pseudoscience to jump in and fill the gaps in knowledge with answers that lack supporting evidence but which are dressed up convincingly with enough scientific vocabulary, that it sounds plausible to the average reader.

It took me years of study and a university degree to fully be able to research a research study and spot inconsistencies and fallacies within a piece of work or product. Being able to critique a research study isn’t just useful for eliminating pseudoscience but it’s useful for evaluating any study that promotes itself as a scientific study worthy of publication. Many individuals still make the mistake of taking a study at face value, particularly if it mentions the words “scientific study” by a “doctor” regardless of said doctor’s field of expertise In relation to the research topic. Media outlets still insist on dumbing down research results which is problematic if the findings get lost in translation. Fortunately with the age of information, many are now becoming aware that a study is worthless unless we can examine and assess the quality of the study itself.

And this is why “news sources” keep saying studies say drinking nothing but tea will lead to a third foot.

In light of the ever growing pile of unscientific nonsense and pseudoscience promoters plaguing our lives, here are some handy hints in how to separate fact from fiction:

1. Look at the researcher or promoter of the product. Just because someone is a Doctor or has a Ph.D or labels themselves an expert, doesn’t make it so. Look at the credentials of the person who is promoting the pseudoscience product or study. Doctor Alan Cantwell is a long time AIDs conspiracy theorist who has written many books on the subject of how AIDs is nothing more than biological warfare injected into Hep B vaccines yet he’s a retired dermatologist.

2. Look at the sources used in the study. Is the research new? Is it an elaboration on previous research? Have a look at the literature review. The purpose of a literature review is to examine the current body of research around a particular topic, explain the current research findings and provide a context for the reader to establish where the study fits into the current field of work around a topic. Without a literature review, the reader will have little knowledge into where the research is coming from and how the research came to be. Literature reviews are good for establishing if the topic of the study has been examined before and what previous studies have stated.

Further to that, where are the author’s sources coming from?  If the author is making a claim, it’s not only vital to assess the claim but the source of where that claim came from. Is the source a quality research paper? Is it a website or a book? If so, how creditable is it? If someone tells you the sky is pink, not only should you do your own research into the pinkness of the sky but also examine the quality of where that claim came from – did the claim come from a meteorologist or from my next door neighbour who seems to think chocolate is how the devil enters your body?

3. Ask yourself, why is the research being studied? Examine why the author is doing the study and what’s in it for them. Do they have an agenda? Are they going to gain anything by writing a study on how drinking nothing but tea will lead to spontaneously growing a third foot? In addition to examining the author’s sources, the funding sources of the study must be looked at. Perhaps the study has been funded by a coffee disturber wanting to discredit tea for its own gain? Are the funding sources impartial? Are the researchers themselves impartial to the study?

4. Does the language make it look like it’s trying too hard? If a study cites that a certain amino acid in shampoo can make your hair shiny but can’t or doesn’t explain how or even what amino acid they use – even in simple terms – it’s probably trying too hard to look like legit science by using the occasional fancy sounding buzzwords. Further to that, examine the specific language of the article – does it use emotional language? Does it use scare tactics by using scientific terms without explaining them in simple terms?

A good example of this is the famous “dihydrogen monoxide” hoax created by Messers Eric Lechner, Lars Norpchen and Matthew Kaufman as a joke. It was then revisited by Nathan Zohner in his science experiment “How Gullible Are We?” to highlight how panic can spread by using emotional language, scare tactics and exploiting a person’s inability to research.

Also does the study use vague language that can never be explained in depth? Words like “toxin”, “holistic”, “detox”, “chemical free” and “natural”.

5. Are the claims realistic? What sort of claims is the study making? Are they realistic or is the study telling you that by eating an orange a day, you can reverse the aging process and develop telekinesis? One example is to type “Stem Cell Cream” into Google and examine the results.  My first result exclaims “Derived from adult stem cell with 387 growth factors”. Funtimes.

My personal favourite is when fad diets pop up on my Facebook promoting themselves with a testimonial from a woman who last 15kg’s in 2 weeks. Funnily enough they don’t like it when you respond “If you lost 15kg’s in 2 weeks, you probably should see an oncologist”.

6. Does the claim rely on testimonials to back it up? Pseudoscience has a hard time backing itself up with empirical evidence – generally because there is none.  So in the absence of empirical evidence, pseudoscience relies on testimonials that are often written in such a way to make us believe that the product or service completely revolutionised the writer’s life. Remember kids, anyone can write a testimonial.

Example: Before I discovered peppermint tea mixed with orange juice, I was a jittery coffee drinking mess. But thanks to its great taste and mixture of amino acids and care bear love – it completely changed my life. I no longer wake up feeling tired and sluggish and my telekinetic abilities have really improved. I can even run marathons thanks to my third foot! Thankyou peppermint tea and orange juice. You changed me for the better.

Authors note: Peppermint tea and orange juice tastes like what I assume battery acid tastes like.

7. Has the study been designed correctly? When executed, a poorly designed study can present results that are less than reliable. Has the study accounted for any extraneous variables? Has the study attempted to reduce researcher bias or the placebo effect? Is there a control group? Can the study be replicated in a clinical setting and can we expect the same results by replicating the study?

8. Does the study or product claim exclusivity or a worldwide conspiracy? If a person makes a claim that they and only they have figured out the cure to a disease that medicine is still working on – chances are they’ve been dipping into some battery acid a bit too much. Look for key terms like “secret” or “things doctors don’t want you to know” or “you need to wake up!”.  If the product makes claims to a worldwide conspiracy involving doctors, scientists and pharmaceutical companies, please click that back button on your browser. Run away fast if the pseudoscience promoter you’ve encounted pops a headache tablet while diatribing about how “Big Pharma” give little Timmy autism through a vaccine promoted by evil doctors.

More red flags include the product or study claiming to cure illnesses or disorders or multiple illnesses or disorders that currently have no cure. A glowing red flag is if those multiple illnesses or disorders are unrelated. Naturopaths for example have been known to claim that their remedies can cure everything from Bipolar, Depression, Autism, ADHD, Restless Leg Syndrome, IBS and Dyslexia among others.

9. Does the study rely on confirmation or refutation? Pseudoscience loves to make grandiose claims about itself. Pseudoscience also loves to make assertions that don’t allow for the possibility that they can be shown to be false by way of an experiment or observation. This is called falsifiability. In other words, why science accepts the notion that an experiment could be designed that could refute its claims, pseudoscience does not allow this.

The largest red flag by far is the use of the logical fallacy Argument from Ignorance – in which a promoter of pseudoscience or study claims that if the product or study cannot be rendered false – this automatically makes it true. This is commonly deployed alongside of the “Reverse Burden of Proof” tactic. While the Burden of Proof is on the person making the claim – pseudoscience loves to reverse the tactic as a way of discrediting the skeptic.

10. Does the study of promoter make unprovable claims about individuals who disagree with them?  Often this tactic is aimed at individuals who are advocates against the product or study. One notable example is the anti-vaccinating community and their numerous claims against Doctor Paul Offit. In 2011, Meryl Dorey of the Australian Vaccination Network asserted the Dr Paul Offit still continues to receive revenue from a patented vaccine after it was sold – this claim was later rendered false. The tactic is a favourite to discredit any opposition to pseudoscientific claims due to the delusion that discrediting any opposition somehow makes the pseudoscience more creditable.

Being a skeptic I often feel like an outlier on the bell curve that is life. In the modern age, the term skeptic has seemingly morphed from one of “healthy investigation” to “close minded opposer of All The Things Ever”. Being a skeptic doesn’t mean that we are closed minded individuals intent on bursting the bubbles of the masses, it means we need a little more than “Hey just trust me , this really works” before we shill out $150 for a product labelled “Tears of Homophobes – Guaranteed to make you 100% more attractive to the same sex”. Being a skeptic means we are less likely to be fooled into pouring money into battery acid teas or homeopaths with questionable office locations.

Being a skeptic means we take nothing on face value because the world is too grand and wonderful to not investigate it thoroughly. I want to find out All The Things, not sit back and be content with subpar explanations with no evidence.

So be a skeptic and hopefully using the above tips – more people around you can be skeptics to.

– Rayne

If you like some of the things I say – I now have a Facebook page! Feel free to like my page by clicking here

Tags: Science

About the Author

Rayne

Related Posts

Evolution – it’s a thing! →

22 Things We Should Be Saying to Mums Who Don’t Vaccinate: Part 2 →

Rejecting Modern Science: Why the mummy instinct is not enough →

Appeals to Nature: The one fallacy to rule them all →