How far can you trust a scientist?

Our chief consultant writes: It’s time to take up the question again of how a non-expert can evaluate an expert. In a previous post I introduced paradoxers, people from outside the field who claim to have made some highly important advance. Paradoxers are interesting in several ways and we’ll return to them later. Today we consider the case of some undoubted expert, a renowned or at least established scientist, describing something in his or her field. How much of it can you rely on to be true? How far can you actually trust a scientist?

This subversive-looking question occurred to our astronomer a while back. To try to answer it, he looked into the specific case of astronomy over the period 1833-1944, and wound up writing a well-regarded book with his conclusions. It’s 300 pages long, so I’m not going to try to include the whole thing in this post. His depressing conclusion is that even the best and most conscientious scientists may make statements as known fact, that turn out later to be wrong. His encouraging conclusion is that this is very rare, and for almost all scientists almost all the time, their presentations are reliable. In addition, there are a number of clues that help indicate the reliability, or otherwise, of what you’re being told.

As a preliminary, you should understand that science is really a process, not a body of knowledge or techniques. In a mature science the process has produced a collection of results that are firmly grounded and completely trustworthy; another collection that appears quite likely and forms the basis for further progress, though there’s a real chance it may have to be changed if something new pops up; and a third collection that contains the ideas now being tried out and tested. (The three types of result shade into each other, of course. The division given here is just one way of classifying them.) The biggest problem, our astronomer found, occurs when something in the third collection is presented as if it were in the first.

Nowadays, there is more pressure than ever to do this. It is the new ideas and speculations that are most exciting, and therefore the ones picked up by the media. A possible new cure for cancer is widely reported; when it doesn’t work out (and most don’t) you don’t hear about it. (I’m not simply being negative. See the article Journalistic Deficit Disorder). Confirmation of a well-known theory isn’t exciting, but something that may overturn it is. You needn’t suspect some vast conspiracy among the editors and reporters: it’s just a direct result of the fact that a media-related company that fails to attract readers/listeners goes out of business.

So, what can you do? Well, our astronomer suggests that when you meet up with a piece of exciting popular science,

• Be aware that there might be a problem. Almost always, everything’s fine. But keep in the back of your mind the possibility that speculation is being served up as fact.
•  Pay attention to the caveats. Scientists add things like, “as far as we know,” “within the accuracy of the measurements,” “probably.” If you keep these in, their statements are true. If you strip them out, the statements often become false. The media routinely strip them out because they detract from the excitement. If you can find what the scientist actually said, it’s much more likely to be true.

Our astronomer also noted that even a little knowledge is useful. If you can remember something from your sophomore Physics class (assuming you remember it correctly!) you can go a long way toward checking what you’re being told. A bit of skill with algebra and numbers is even more powerful, if you’re given something along those lines to work with.

If you’re faced with something important and it requires some high-level skill in physical science or mathematics to sort out, of course we’d be happy for you to use our Evaluation service. But the more you can do on your own, all along the way, the better off we’ll all be.

The next part of this How can you tell? series will look at the hardest situation of all: when there are two sides to an argument, each supported by (apparent) experts.