Of the Evaluation of Expertise ("I am not so good for that as an old roofer")

Attention conservation notice: A 2000-word reaction to conferences in 2011 where too many people wanted impossible things of social science and data mining, and too many people seemed eager to offer those impossible things. Too long by more than half, too pleased with itself by much more than half, it lacks constructive suggestions and even a proper ending. To the extent there's any value to these ideas, you'd be better off getting them from the source I am merely parroting. Left to gather dust in my drafts folder for years, posted now for lack of new content.

Q: You have an old house with a slate roof, right?

A: You know that perfectly well.

Q: Does the roof ever need work?

A: You just said it's old and slate, and I live in Appalachia. (In the Paris of Appalachia; but still, in Appalachia.) Of course it does.

Q: Do you do the work yourself?

A: I have no idea how; I hire a roofer.

Q: How do you know the roofer knows what he's doing?

A: I am not sure what you mean. He fixes my roof.

Q: Well, does he accurately estimate where leaks will occur over the next year?

A: No.

Q: Does he accurately estimate how much the roof is going to leak each year?

A: No.

Q: Does he accurately estimate how many slate tiles will crack and "need replaced" each winter?

A: No.

Q: OK, so he's not much into point forecasts, I can get behind that. Does he give you probability forecasts of any of these? If so, are they properly calibrated?

A: No. I don't see where this is going.

Q: Everyone agrees that the ability to predict is a fundamental sign of scientific and technological knowledge. It sounds like your roofer can't predict much of anything, so what can they know? You should really hire someone else, preferably someone well-calibrated. Does Angie's list provide roofers' Brier scores? If not, why not?

A: I don't believe they do, I can't see why they should, and I really can't see how knowing that help me pick a better roofer.

Q: It's your business if you want to be profligate, but wouldn't it help people who do not enjoy wasting money to know whether supposed experts actually deserve to be taken seriously?

A: Well, yes, but if you will only listen to roofers who are also soothsayers, I foresee an endless succession of buckets under leaking ceilings.

Q: So you maintain a competent (never mind expert) roofer needn't be able to predict what will happen to your roof, not even probabilistically?

A: I do so maintain it.

Q: And how do you defend such an obscurantist opinion? Do you suppose that a good roofer is one who enters into a sympathetic human understanding with the top of the house, and can convey the meaning of a slate or a gutter?

A: Humanist-baiting is cheap even for you. No, when it comes to roofs, I am all about explanation, and to hell with understanding. (People are different.) But an expert roofer no more needs to predict what happens to the roof than an expert engineer needs to predict how a machine they have designed and built will behave. Indeed, it would be a bizarre miracle if they could make such predictions.

Q: And why would a plain, straightforward prediction be such a wonder?

A: However expert the roofer is about the roof, or the engineer about the machine, what happens to them depends not just on the object itself, but the big, uncontrolled environment in which it's embedded. You will allow, I hope, that what happens to my roof depends on how much rain we get, how much snow, etc.?

Q: Not being a roofer, I don't really know, but that sounds reasonable.

A: Might it not also matter how many sunny days with freezing nights we have, turning snow on the roof to ice?

Q: Sure.

A: And so on, through contingencies I'm too impatient, and ignorant, to run through. But then, to predict damage to the roof, doesn't the roofer need to not only know what condition it started in, but also all the insults it will be subjected to?

Q: That does seem reasonable. (But aren't you asking a lot of questions for "A"?)

A: (Shut up, I explain.) So your soothsaying roofer must be a weather-prophet, as well as knowing about roofs. And the same with the engineer and their machine: they would need to foresee not just the environment in which it will be put, but also the demands which its users will place upon it. It sounds very strange to say that such prophetic capacities are a necessary part of expertise in roofing, or even engineering.

Q: It might sound strange; many true things sound str