My PhD dissertation was about bias in cost and ridership forecasts for transit projects. Before getting into any data analysis, I address the question of how we should even be evaluating forecasts in the first place. One response to evidence that forecasts for transit projects have generally proven to be overwhelmingly biased has been an argument that forecast accuracy is unimportant, or less important than other considerations. And it’s true that accuracy isn’t the only possible way to evaluate a forecast.
A 1993 essay on weather forecasting by Allan Murphy (which I came across by way of Nate Silver’s book The Signal and the Noise) defines forecast “goodness” in terms of three characteristics:
(1) Consistency: Is the published forecast consistent with the forecaster’s best judgment? Does the forecaster actually believe her own forecast?
(2) Quality: Does the forecast correspond with what actually occurred? Was it proven to be accurate?
(3) Value: Is the forecast useful to forecast users? Does it help them to make the best decisions?
Individual forecasts might be good in one or more of these ways, without being good in all three. For example, a financial forecaster might try to defraud investors by intentionally inflating her firm’s earnings forecast, but unexpected events occur later that end up making the inflated forecast accurate (thus, the forecast has good quality, but poor consistency). A weather forecaster might intentionally overstate the seriousness of a storm (poor consistency) because she’s knows that people would otherwise under-prepare. Although the storm turns out to be less serious than her published prediction (poor quality), lives were saved because people were more prepared than they otherwise would have been (good value). An economic forecast might be really rigorous and accurate, but so vague and presented with so much technical jargon that it’s ultimately useless to the lay forecast user (good quality; poor value).
In evaluating all of a particular forecaster’s forecasts over time (rather than just one individual forecast), Murphy proposes that the best way to maximize all three types of goodness is to maximize forecast quality. Consistently accurate forecasts will increase the forecaster’s confidence in good methodologies (improving consistency), and the value of a set of forecasts increases with forecast credibility, which correlates with the accuracy of past forecasts.
Murphy goes on to argue that another reason to emphasize forecast quality over other types of goodness is that it’s often the only of the three types of forecast goodness that can be observed at all, since we’re generally no better at reading minds than we are at predicting the future. We don’t know what’s going on in the mind of the forecaster, so it’s hard to really judge how well a published forecast corresponds to the forecaster’s best judgment. Likewise, since we can’t read the minds of forecast users, we can’t really say how the forecast has influenced their decision-making. So we’re left with accuracy.
I think these three types of goodness are a useful way to think about what we might mean when we say, “The Church is True.” We hear and say this a lot in our church, and it strikes me that different people mean and understand different things by it. Applying Murphy’s three types of goodness, we might be referring to:
(1) Consistency: Current and past church leaders and teachers at various levels are sincere in their teachings and truth claims.
(2) Quality: The teachings of the church [fn] accurately reflect reality.
(3) Value: Decisions that follow church teachings can improve a person’s life.
Thinking about which of these three types of goodness should get the greatest emphasis when we’re talking about the truthfulness of the church is a little different than when we’re evaluating forecasts. The quality (accuracy) of most forecasts can be empirically observed, but most church teachings (the nature of God, for instance) don’t lend themselves well to that kind of direct confirmation. We have the same problem with evaluating consistency as we do in the case of forecast evaluation. No one but Joseph Smith (or President Monson) can really know how sincere he was (or is) in his belief in his prophetic calling.
Speaking generally about all people who encounter church teachings, we might run into the same problem with evaluating the value of church teachings that we do with the value of a forecast. We don’t necessarily know how people are using them. However, we can evaluate the value of church teachings on a personal, individual level. For you personally, does the Church work the way you need it to? Are the doctrines helpful to you?
I think scripture verses like Matthew 7:16-20, John 7:17, and Alma 32:27 are an argument that —similar to how a set of forecasts with good quality is most likely to also have good consistency and good value— a set of doctrines that you’ve found to have good value is likely to also have good consistency and good quality. This comes pretty close to how I think about my own testimony and experience with the church. Not that the accuracy of the church’s claims or the sincerity of church leaders are unimportant, but that my own personal experiences with church doctrine and church participation are (first of all) also important, and (second) the closest I can come to any kind of evidence of the other two types of goodness.
[fn] There could be a related, but entirely separate discussion about what “the teachings of the church” include or don’t include. I’m choosing not to get into that here.