Dueling PhDs

Our chief consultant writes:

We come to the hardest problem to set a layman: suppose there are two (or more) experts, that is, people who disagree strongly about some scientific or technical question, each of which has some claim to expertise. Call this “dueling PhDs.” You, as a layman, are called upon to decide between them. What do you do?

We assume that the matter is advanced or esoteric enough that there’s no question of you actually checking the math yourself. Also, that it’s not a matter of current research, where the answer really isn’t known. (That excludes, for instance, Stephen Hawking and Roger Penrose disagreeing about the Nature of Space and Time.)

You could try to evaluate the credentials of either side, working out whose PhD is stronger; or take of vote of scientists working on the matter; or, perhaps, decide on the basis of motivation, asking who is funded by whom. Slightly more useful is analyzing the rhetoric, on the assumption that someone with a poor scientific case is more likely to try to cover it with noise. All of these techniques leave us uneasy, and though we have some suggestions on how to use them we prefer a harder, more time-consuming method as a more reliable way of getting at something like the truth.

Of course, if the matter is important and falls within our expertise, you could make use of our Evaluation service. But the question is more important than just drumming up business for ourselves. Back to what you can do:

There is a point to scientific credentials. The Chair of an Astronomy Department at, say, a Princeton or Harvard or Cambridge, has gotten there by doing a good deal of science, all of which has been mercilessly evaluated by people well-equipped to do so. Someone holding a less prestigious post has not undergone quite the same level of examination. It doesn’t mean one of the former is always right and the latter always wrong, but if you’re forced to bet that’s the way to go. Looking on a wider scale, you could try to find the consensus of scientists working in the field: if an overwhelming majority of them say that a certain result has been firmly established, they’re almost certainly right.

But, if you’re not actually working in the field, figuring out whose PhD is better or whose institution more prestigious can be impossible. In addition, we’ve known cases of members of even the highest-ranked places whose own science is wrong enough almost to qualify them as “paradoxers.” And counting scientific noses from outside is also difficult, especially in a politicized field: we’ve seen one side of a dispute claim that the other side has only 40 scientists, when the true figure is a thousand times larger.

You could work out the balance of suspicion by following the funding trails, which are informative sometimes (a pharmaceutical company? But some drugs are safe and effective). More reliable is to look for smokescreen rhetoric: accusations of a cover-up conspiracy among scientists or “the establishment” very often are there to draw attention away from a poor scientific case. If Galileo is mentioned that’s an ominous sign.

But using any of these clues to decide which “expert” is right makes us uneasy. Science should not depend on who does it, or why. Even an unpalatable character can come up with the right answer sometime. So we suggest, if you have to weigh expertise out of your own field, the following procedure:

First, ignore any summary or set-piece debate in the mass media. Find an exposition of each side or each expert in their own words. Not only does this get around any possible misunderstandings or misrepresentations of third parties, it can give valuable clues as to the experts’ own thinking.
Second, dismantle any rhetoric, talk about motivation, accusations of character, anything personal. Science is done by scientists, who are human; but we want to deal only with the science.

What’s left should be an exposition of a bit of science. If it’s muddled and confused, you’re entitled to distrust it; if there’s nothing left at all, we’re not dealing with a scientific question. Beware of reasoning by analogy. Analogies can be effective for explaining things, but they prove nothing. “The Sun is like a toaster”—well, yes, in some ways, but not in others, and the Earth is not very much like a piece of toast. (We plan a blog post later on discussing things like this.)

There are indeed people who are very good at constructing plausible nonsense, and good scientists who are not good explainers. But at the very least you should have reduced the “debate” to a question of science, which may be enough in itself to point the way to the answer. You might be able to work it out yourself!