When finding nothing is something

In science, coming up with the answer is often the easy part.

Our astronomer is doing some research.  In this case it takes the form of analyzing a set of data to see if certain things are related to other things; for instance, if the accuracy with which the brightness of a star is measured depends on its color.  This is the “testing hypothesis” phase of the Scientific Method.  Scientists rarely structure their research explicitly around the Method as taught in school, but it does show up in practice.  And there are libraries of books explaining various sophisticated statistical ways to test hypotheses.  Their sheer number indicates that it can be a hard problem.

The main difficulty is when there may be a relation, but it’s not clear and clean.  The danger comes from our human tendency to see patterns in noise, to detect objects that aren’t really there.  (We suppose that it has some evolutionary advantage: the leopard that you don’t see, but is there, can kill you; the one you do see, but isn’t there, is harmless.  However, spending too much time running away from your imagination hurts your hunting-and-gathering, so there are limits to the tendency.)  Hence ways to tell objectively if there’s signal in noise are important.

That’s not our astronomer’s problem at present.  In several cases his results are clear: there’s no relation.  His scatterplots are well and truly scattered.  The jungle is clearly bereft of leopards.

That in itself is interesting.  Several of the hypotheses were reasonable and plausible, and indeed at least one was common wisdom.  But now the hard part begins: how well do we know there’s nothing there?  How big an effect could there have been, and we not see it?  This can be a vital question, and answering it is generally harder than finding the result in the first place.

Students in Physics classes are finding this out every term (at least, their teachers hope so).  Over and over again experiments to determine, say, the acceleration of gravity are performed by many students.  It’s not because the value is unknown; it’s known very well.  It’s to teach the students to work out how well they know their answer.  The simple idea of uncertainty, that the number is this plus or minus a bit, is central to science.

And when the answer is zero, it’s really an upper bound.  Something too small for us to see could be present in the data.  In that case, knowing how big it could be is the important result.  Is it possible that the jungle is hiding a leopard cub?