What are some characteristics of a good forecast?
A Good Forecast is Useful
Whether made by reading tea leaves or calculating on supercomputers, a major motivation for forecasting has always been gaining a degree of control over the future. If you know what the future will bring, you can plan more effectively to deal with it. If certainty isn't possible (and it rarely is), partial knowledge, or even just knowing how the odds of certain scenarios stack up let you make more informed decisions than you would without this information. There is little point in making forecasts, even if they are 100% accurate, if they do not provide information that you do not already possess -
The accuracy of a forecast also contributes to the utility. Incorrect forecasts may be quite costly, depending on how they were used and what decisions were made based on them. The relationship between value and accuracy, however, is complex and is highly dependendent on the forecast's actual application and the stakes riding on the downstream decision-
A Good Forecast is Non-
A Good Forecast is Specific
When you make a forecast, you make a statement about the future. A forecast that is sufficiently vague that it can interpreted in multiple ways, especially in hindsight, may be less than useful. The classic example of this isf when a psychic predicts that a certain celebrity will experience a 'traumatic event' in the 'near future'. The prediction is sufficiently vague that it can be 'validated' by any number of future events that may occur.
The when of a prediction is also an important part of specificity. A good prediction also specifies some kind of timeframe for a predicted event ot occur in. The prediction that you will die is trivial in that barring amazing medical advances in the near future, it represents a certainty. Adding detail such as 'dying in a traffic accident' or 'dying on February 22nd' adds specificity. The principal value of specificity isn't that more detail is better in itself, but more detail can often make the forecast more useful.
Sometimes forecasts are made in the form of probabilities, such as a '40% chance of rain tomorrow'. This kind of forecast should not be viewed as a hedge, but as recognition of uncertainty, and an attempt at quantification by the forecaster. In addition to signifying confidence, probability estimates in forecasts can also serve as key inputs to planning and decision-
A Good Forecast Has the Possibility of Being Wrong
This may sound completely oxymoronic, as people generally find incorrect forecasts to be of little or no value, or sometimes even very costly. My reason for writing this is that a 'good' forecast may be viewed as a kind of scientific hypothesis. Karl Popper, a philosopher of science, maintained that a critical part of a scientific theory is its ability to be falsified by contrary evidence. For example, by Popper's criterion, the statement 'Space aliens may have visited the Earth' is not scientific while the opposing statement 'Space aliens have never visited the Earth' is. The first statement is not unscientific because of the opinion expressed, but because it is practically, and in principlel, impossible to disprove. A lack of evidence of extraterrestrial landings just means that you have not been able to find any -
The latter statement, however, can be readily disproven if someone can present credible evidence, such as a spacecraft or an actual alien. This ability to prove the statement false, and the related ability to define what would consititute sufficient contrary evidence to do so is what makes it 'scientific', not the particular opinion expressed.
Similarly, many predictions take the form of statements that can be shown to be either true or false after the predicted event either occurs or fails to occur. For example, a prediction of which team will win a football game can be readily verified by looking at the game's final score. Saying that the game could go 'one way or the other', isn't a forecast.
Some predictions, however, may not have binary outcomes, but numerical values or expected ranges -
A Good Forecast is Made Through a Credible Process
HOTSPUR: Why, so can I, or so can any man, But will they come when you do call for them?
Henry IV Part 1, Act 3, Scene 1, William Shakespeare
Much as anyone can 'call spirits', anybody can make predictions by any number of means. While you are likely to consistently fail at spirit-
A forecast can be compared to a decision in the sense that it is possible to separately consider the quality of the decision-
Although important, defensibility is only one criterion for determining if a forecasting process is any good. In cases where the forecast is made many times, the track record can be a far more important factor. This is especially applicable to statistical and machine-
21 NOV 2016