The Scientist’s Dilemma

“Extraordinary claims require extraordinary evidence.”

This iconic quote, popularised by Carl Sagan, resonates throughout human life as we navigate the sometimes scintillating, sometimes hazy world of knowledge that bombards our senses each day. The world is a jostling marketplace full of ideas vying for a place in our minds. How can we hope to filter it all into a viewpoint that makes sense?

Let’s look at an example. We generally accept simple claims at face value. If I (a 23 year old) said “My mother is 47 years old”, it is quite believable. It fits with widespread experience; an age gap of anywhere between about 16-35 between mother and son wouldn’t raise too many eyebrows. I doubt that the average person would demand to see documentary evidence.

Change the statement, though, and we see a different story. I’m sitting opposite you in a pub, sipping a pint of ale. I look you in the eye and say “My mother is 4 years old.” You’d laugh (or talk to someone else) but you wouldn’t believe me. If I insisted it was true, you would demand evidence. Extraordinary claim? Let’s see your extraordinary evidence.

The statement is useful when dealing with all kinds of outlandish claims, from some alternative medicines to the more extreme religious viewpoints. However, it also applies in our everyday lives, and it is especially important for scientists.

Imagine I’ve discovered a negative correlation between squirrels and heart disease rates in London – the more squirrels in an area, the less heart disease people suffer. To have this work recognised by my scientific peers, I need to convince them it’s both new knowledge and supported by evidence. If I come up with something mundane and already known, no-one would care and it wouldn’t get published. So, to meet the first criterion, I need to show how interesting my idea is, and why it’s different from what has come before.

In this case, it’s obviously new – no-one’s thought to examine squirrel distributions against heart disease rates! And, in my imaginary world, the correlation between them is strong. I boldly propose my idea: living in an area where you see lots of squirrels will lower your risk of heart disease.

Nom Nom Nom.

Aha! But if the work is unique, and the idea is a major one, then it’s an extraordinary claim. People will demand strong evidence! Hard scientific evidence is tricky to gather and present – especially to prove causation between squirrels and less heart disease. To deal with this, I need to make sure my idea is new, but also draws on established scientific work. If I explain my finding in terms of what other people have already found, it looks less extraordinary; the burden of proof is eased.

For example, I could point to existing evidence that connecting with nature makes people less stressed; or that squirrels might prefer to live in areas with cleaner air or more open spaces where people are drawn to exercise, or some such. If I can find studies which either say, or imply, such things, it will be easier for people to accept MY new claim.

Therefore, I am faced with a dilemma. For people to notice my work, it must seem new and important. However, the level of evidence to make people believe me is relative to how new and different the work is from what has come before. If it’s a big, new discovery, I need lots of evidence; a more mundane claim, which is closer to existing work, can get away with less. I call this the ‘Scientist’s Dilemma’, faced when publishing or communicating new research.

An excellent example of the ‘Scientist’s Dilemma’ gone wrong was the Arsenic Alien Bacteria fiasco late last year. For a blow-by-blow of what happened, check out Ed Yong’s summary at Not Exactly Rocket Science. In short, the scientists and initial media promotion fell into the trap of making an extraordinary claim without the evidence to back it up, and the backlash was swift and hard. However, without the media push, the paper may have been doomed to obscurity.

I like the concept because it’s something everyone, not just scientists, can relate to. We all love telling stories and sharing ideas with other people, whether it be on a blog, out in the surf, with the local Dungeons and Dragons society or at the pub. Every time we start telling a story, if we want people to believe it, we need to keep the dilemma in mind: the more extraordinary and interesting the claim, the harder it will be to make our audience believe. Science simply takes this one step further, adding a formal structure and rules to the game of storytelling we all play.

Edit: Welcome to all those arriving from Freshly Pressed, thanks for the visit and feel free to look around! As it happens, I’m currently in the running for Best Science Blog for 2011; I’d love it if you could help me win by taking a minute and voting! More details and how to vote are here!

Advertisements
Categories: Problems, Science, Thoughts | Tags: , | 58 Comments

Post navigation

58 thoughts on “The Scientist’s Dilemma

  1. Great post. So does anyone think that the idea of “Extraordinary claims require extraordinary evidence” can also speak to the concepts of love and marriage?

    I do.

    ;)

    • J Roycroft

      I have an extraordinary claim as it relates to my marriage. My wife is half my age, pregnant, and we have a beautiful little 2 year old girl. Being married to a younger woman definitely adds to my life expectancy. My ex wife could kill me and probably would if she ever got sober. I wake up every morning with a grand smile on my face. Life is good in the Roycroft family. Oh and I do like to watch the squirrels play. Just sayin!

    • Yes, I think it doesn’t have to just be limited to the physical world around us! If some random stranger off the street ran up to us and declared undying love, we’d think they’re crazy, but if you have the evidence of attention, friendship and long-term devotion, and that special extra something, then a statement of love seems very right!

  2. vanimator

    Great Post !!
    Thanks for the share.

    Vanimator
    http://www.vanimator.com

  3. J Roycroft

    Congrats on FP

  4. Uh oh, I live in Arizona and I rarely see squirrels …

  5. Good quote, and it should apply to all… but i can’t help but wonder if it does.
    With all the misinformation that i see lately, and all the crazy claims that are readily believed en masse, i have to wonder if this is a novel sentiment that is well accepted, or merely should be.
    Either way: cool, topical, and relevant, so thanks for sharing,
    ~J

  6. Agreed. But have you ever noticed how sometimes people turn away great ideas simply because the basis of the ideas already existed? They’ll probably be like, “ANYONE could have done this, so it isn’t such a great idea after all.” Even though ANYONE has probably never taken the effort to do it, but the person with the idea has.
    Congratulations on being Freshly Pressed!
    Ashley, aka TheEverydayMuser
    http://www.theeverydaymuser.wordpress.com

    • Yes, and that’s a problem with science; often, simple assumptions can go untested, simply because they are so basic. It isn’t always just because scientists don’t want to look at them, though, it’s that even simple research takes time and money, and grants for looking at simple, obvious things are hard to come by!

  7. I love how you explain the concept in regards to science then show how it applies to most other areas of life. It reminds me of trying to come up with a good idea for a movie script. It has to be new, different, and exciting while being something believable people can relate to.

    • Great example! It’s a real bind – if it’s too unfamiliar, you’ll lose your audience from the outset, but if it’s too similar, you’ll get rejected or ignored because it’s boring. I’ve been taking some classes in media and TV and have heard exactly the problem you’re describing.

      • That IS a great example. So much of entertainment is very familiar, and I wonder about the decision processes of those who agree to green light the projects.

  8. anonnickus

    Wonderful posting from top to bottom. If you make a scientific break-through and no one publishes it is it a break-through? Good question. When you ask the correct question it is the answer that is pointless. I love squirrels and I loved your post and I hate heart disease.

  9. Love your scientific mind.. lol… usually my extraordinary evidence to my tales is: “I once heard of a…. or there was this thing….”. West London has some extraordinarily big squirrels… hmm….

  10. Great post, thank for sharing :)

  11. *tries to think of an original comment* *fails miserably*
    Your blog is AWESOME, congratulations on Freshly Pressed!!! xD
    http://fatalisticoptimism.wordpress.com/

  12. Great post…love your site!

  13. I don’t think you always have to have the proof to back it up, because sometimes the concept itself can be interesting enough to start a discussion. (There are things “seen” and “unseen.”) If people can relate to it in some tiny way, that’s enough to start an interest or spark to ponder. The level of thoughts (thinking) and talk can be enough to get something started. The trick, it would seem to me, is to keep the discussion going. At least, till you can generate enough interest to get the backing you need to for futher exploration (if it’s a scientific endeavor, anyway).

    Yes, we are constantly bombarded with ideas and knowledge (Isn’t it wonderful?!), but you start to sift through it all as you grow, looking towards the ones that peak your own interests, and you turn the rest off. And there’s always meditation and vacations to help us with that sort of thing.

    I like your squirrel example. It reminds me of looking at fish in an aquarium and the effects it has on people’s health.

    • That’s true. There are some ideas which can be really rewarding to entertain, even if they aren’t attached to anything specifically real or evidence-based. However, if they become a specific claim about something real, evidence becomes important!

  14. Ed

    Really enjoyed the post, a witty example of a problem all of us scientists go through,e specially when trying to present our data to the public. Thanks to Freshly Pressed for bringing your blog to my attention.

  15. timcowan64

    Interesting post, love the “universal” wisdom! Although I’m getting stuck on using this as an excuse to sit around with a cup of coffee, squirrel watching… let us know if you manage to prove that premise!

  16. Well written! Congrats on being freshly pressed!

  17. Interesting post David, thanks for sharing and congratulations on being freshly pressed.

    One of the sayings that I’ve always remembered from my school days is that “absence of evidence is not evidence of absence”. I think that this is a part of the scientist’s dilemma in that it helps to continue to push science to the edges of current knowledge and works towards uncovering the evidence for minor and extraordinary claims alike.

  18. paul kiari

    I like this applied to discussion at a pub and with friends, but as a young scientist myself I’m curious as to the role method and rigour play into this. you could have completely shocking data that conflict with all of the past data on the issue and can’t be explained by it, but if you experiment was rigorous then the conclusions will go under further experiment.

    I have not looked at the experiment you mentioned above enough, but as far as i know it is was certain errors in the rigour of the experiment that weakened the force of the extraordinary conclusions.

    basically, the ‘extraordinary evidence’ needed may be forthcoming as well as already grounded in the past. of coure we can’t fully accept such claims, but we can be excited and place an appropriate amount of weight on it.

    this isn’t to say i dont love the article :)

    ps… check out mine blog as well if you’d like paulkiari.wordpress.com

    • Yes, good point, and thanks for the comment! It’s interesting though, I’m not sure how many ‘revolutions’ in scientific thought have come from new evidence, as opposed to a new way of thinking about the evidence.

      That’s not to say evidence can’t throw a whole new light on a subject, and as you say, rigor and reproducibility feed in strongly at that point. The structure of science is designed to ‘sort out’ which explanation stands and falls.

  19. Nom Nom Nom.
    Neces squirrel :)

  20. Nice, sorry

  21. …I’m writing this comment from Jupiter. haha…
    Great post, thanks.

  22. Good point. Sometimes I have to work with statistics but I have to follow somebody else´s rules and analize them the way I´m asked. Even if they are logical, the results are pure nonsense that merely reflects reality. Yet, everybody takes them as good. And I want to scream because they are not using their heads to think, they just believe the chart as if it were a sort of religion.

  23. Your quote “The world is a jostling marketplace full of ideas vying for a place in our minds. How can we hope to filter it all into a viewpoint that makes sense?” could have been the wording Richard Dawkins himself would have preferred in his seminal work “The selfish gene”. Here he defined meme to be the cultural equivalent of a gene. A meme as you mentioned in your beautiful quote competes for space in your mind and certainly shapes one’s Weltanschauung.
    I look forward to reading more of your work!

    • Yes, I find the meme idea to be a really interesting one. It’s a very attractive concept and is almost certainly true to a large extent, though I’m not convinced how far it can be taken in drawing parallels with physical evolution. Thanks!

  24. This is a great post! Even though I am not really into much science, I could really take your reflections and test them out in my own life.

  25. Great post and congrats on being FP! I will take a look around…

  26. fl'ame

    Hi David,
    thanks for this vivid post. I like simple language in relation with stuff that seems to be complex. However, some problems you mentioned are not only a problem for scientists. The whole, global society and thus, humanity suffers under a lack of communication. Taking your example:
    Imagine 10 people on Earth who are interested in explaining the causation between squirrels and less heart disease. Imagine 100 further people who are experts in scientific fields you need to consider to prove your thesis. Without intelligent organization of information and consequently the lack of communication: You are alone, finally.

    My thesis: It is not a scientific problem. It is a communication problem in combination with an existing lack of awareness about truth.

    If you want more, simply read http://globalsocietyblog.wordpress.com =)

    • I take your point. There is so much information being generated and already out there that, even if a claim is important and is supported by compelling evidence, the communication of it matters just as much as its support; as a commenter mentioned earlier, if a breakthrough (or idea in any field) isn’t communicated to anyone, then what does it matter?

  27. Fascinating. One of the toughest issues right now is how illiterate so many of us non-scientists are — which allows for political “debate” that is often shrill, uninformed and shaped by people with deep pockets and vested commercial interests, not necessarily the most persuasive data. This must drive you mad!

    As a journo and author, I know the sharp elbows one needs to jostle your way into the marketplace of ideas and try to be heard, let alone quoted, read, reviewed and respected.

    Good luck!

    http://caitlinkelly.com/

  28. I live in “squirrel country” here in Kansas City. We can’t plant anything or grow anything without them eating it. Not sure how that might help our life span, but this was a great read nonetheless.

    Thank you!

  29. I like the idea you started with regarding the nature of accepting claims at face value. I think one of the largest issues we have within our society is the inability to think critically about a situation (like the example of the Arsenic Alien Bacteria you mentioned). Your stated scenario of age difference between a mother and son is a glaringly obvious situation people should be skeptical of. However, when people read from respected news outlets, should they be skeptical in the same manner? Obviously, we live in a “hazy world of knowledge that bombards our senses each day” and should think accordingly given a specific situation. Again, if the majority of society was capable of sound critical thinking skills these types of issues would not be as prevelant.

  30. I’ve never seen very much sense in the “extraordinary claims require extraordinary evidence” idea. It seems to me that ordinary claims are the ones that should require extraordinary evidence. We all absorb a certain amount of basic knowledge from the society around us. This becomes the “ordinary” stuff from which “ordinary” claims arise. However, unless we are selfish enough to believe that our society happens to get everything right, much of this ordinary stuff is wrong.

    There was a time when connecting bumps on the head to personality traits was ordinary. There was a time when the atom being indivisible was ordinary. There was a time when the ulminiferous aether was ordinary. Folks should have demanded extraordinary evidence for these ordinary claims.

    Similarly, today it’s ordinary to claim that prozac cures depression. However, it’s probably not true. It’s ordinary to believing that selecting low-fat foods will make you healthier, but probably not true. It’s ordinary to choose local foods because they’re better for the environment, but it’s not true in all cases. Those are the type of claim for which we should demand extraordinary evidence.

  31. Ideas or sharing an idea to other people is often a success, and yet it becomes unsuccessful when other doubt about it. Oufcourse developing the idea can sometimes be a tough job – just like presenting it in a way than other accept it to be real. If people believe or not, there is always more to thoughts than just does it work , is it possible or how much does it cost – and that’s what science is all about for me anyway……. Just like the author of the post talked about squirrels and “alien” bacteria , I can talk about picture of clouds that look like UFOs. If people believe or not is often decided from the level of understanding the issue or the discussion and the topic. For me it’s not hard to consider the possibility of less heart attacks “due to the” squirrel presence in certain areas. But on the other hand i might see UFOs on the photo, when there was actually nothing more than clouds on the sky. Could be , maybe it is …….. Evidence is what decides on the end, and the percentage of people who agree or support it. And just for the end, the final thought – maybe sometimes people believe without real evidence, because they can relate to the situation, because they understand the situation .

    • Laterzman, your final thought hits the nail on the head. This is especially true when the evidence is incomplete, or complicated, but really we believe many things on trust or for other reasons beside evidence.

  32. Congrats on FP… Liked the concept…

  33. Good post. I think that is there a good correlation between claims and prof, because is there is someone come with something way different, that mean that we are more ignorant about that topic, then we need more info(evidence) about it, so we have a better background, to see if it could or not be true.

  34. faerylandmom

    Excellent post. Congrats on FP!

  35. Well written and illustrated. The simpler the explanation, the deeper the understanding of the subject by the one explaining it. Your blog is logic. Some scientists try to complicate the study of things around them because the have not the desire nor will to actually have their work challenged by laymen or logical thinkers.
    I marvel at scientific programs and publications that say something akin to “Eventually the fin became a wing.” Indeed an extraordinary claim – now lets just go on because only scientist would understand the extraordinary evidence anyway.
    Thank you, young sir for illustrating and extraordinary concept simply.
    Kenton Lewis

  36. Great post… j’aime beaucoup… Mi piace molto (i love it)….

    Best regards…

  37. It’s a great post. And i like the line of thought. I also study science. Physics to be precise.

  38. Hi! I’m new here. Squirrels = less heart disease, really! Then my heart must be in fantastic shape! I’m surrounded by squirrels! I’ve got tons of tiny holes in my yard to prove it. LOL! Great post! :)

  39. Nice article! I used to write a lot more about science and evolution. I think this article is going to encourage me to do that more often.
    Congrats on FP.

  40. This worries me. My cat has a squirrel fetish and is killing them off one by one…….

  41. Great post. Thanks for sharing.

  42. Thanks to everyone for your feedback on this! Some really interesting discussion. And I’m glad I’m not the only one who is concerned for the plight of squirrels around the world. Nature’s healers, that’s what they should be called.

  43. Pingback: Sunday Science: Some of My Favourites | David Robertson

  44. Thanks for Sharing Such a Wondorful Post.

  45. Yes, I think it doesn’t have to just be limited to the physical world around us! If some random stranger off the street ran up to us and declared undying love, we’d think they’re crazy, but if you have the evidence of attention, friendship and long-term devotion, and that special extra something, then a statement of love seems very right!

  46. Pingback: My 5 best posts of 2011 « David Robertson

Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: