It Worked Three Times…

Out of how many?  That should always be the next question.  If your paycheck was deposited successfully into your bank account only once a year, I hope the conclusion would not be “Direct deposit works.  It just works one out of twelve times…”

Returning to the basics, it’s important to remember this definition, from the Merriam-Webster dictionary:

Hypothesis – a tentative assumption made in order to draw out and test its logical or empirical consequences.

Perhaps the most important word in that definition is tentative.  A true hypothesis will evolve as data is obtained and analyzed.  The hypothesis is what changes, not the data.  The big cases of scientific misconduct get plenty of coverage – yet it’s the day-to-day misinterpretations of data that can be the most dangerous.

For starters, we need to be aware of our definition of “worked”.  It’s very common to hear someone say, “It only works one out of every five times I do it- I’m not sure what’s going on.”  It may be true that the result we’re hoping for only occurs one out of five times, but that’s not the result were getting.  The experiment “worked” four out of five times – it’s just not the way we wanted it.

Marrying ourselves to a hypothesis can lead us down a very dangerous road.  Personally, I have wasted plenty of time hammering on an experiment because I just knew it had to work a certain way.  In the end, the finding was the same as what I should have concluded after only a few experiments.  Aside from wasting time, these experimental blinders can significantly limit our ability to objectively analyze data and move the project in the right direction.

This is not to say that trying experiments a few different ways is not worth it.  Far from it – troubleshooting is an essential part of science.  However, it should always be carried out while being mindful of the hypothesis and the data.

If an experiment “worked” three out of ten times – it didn’t work. To simply ignore the other seven experiments is just irresponsible as a scientist.  If it’s a true result, it should be reproducible 10 out of 10 times.  I’ve seen first-hand the damage that can be caused by selective figure presentation and storytelling.  If you are a scientist of integrity, which we all should be, it’s important to look at the results as a whole.  Although this may require reworking of the original hypotheses, it’s the right thing to do.

You give your paycheck a pretty objective look every month, why should your data be any different?