Forecasting - Nature of a Good Forecast - EdsCave

What are some characteristics of a good forecast?

A Good Forecast is Useful

Whether made by reading tea leaves or calculating on supercomputers, a major motivation for forecasting has always been gaining a degree of control over the future.  If you know what the future will bring, you can plan more effectively to deal with it.  If certainty isn't possible (and it rarely is), partial knowledge, or even just knowing how the odds of certain scenarios stack up let you make more informed decisions than you would without this information.  There is little point in making forecasts, even if they are 100% accurate, if they do not provide information that you do not already possess - and can employ to inform your decisions about how to deal with the future.   The idea of utility is central to the forecasting enterprise - otherwise forecasting becomes a mere game.

The accuracy of a forecast also contributes to the utility. Incorrect forecasts may be quite costly, depending on how they were used and what decisions were made based on them.  The relationship between value and accuracy, however, is complex and is highly dependendent on the forecast's actual application and the stakes riding on the downstream decision-making process.

A Good Forecast is Non-Trivial

Making a forecast that the sun will rise in the morning  (despite visibility through cloud cover) is likely to prove accurate, since this event has occured each day for the past several billion years.  The problem, however, is that this prediction is trivial, in the sense of providing no novel or actionable  information.  The opposite prediction -  that the sun won't rise tomorrow, would be non-trivial, despite being highly unlikely to occur.  Although the latter forecast might be quite actionable (what would you do at the end of the world?), the likelihood of occurence, however, is so low that taking action based on this forecast would probably not be a bright idea.

A Good Forecast is Specific

When you make a forecast, you make a statement about the future.  A forecast that is sufficiently vague that it can interpreted in multiple ways, especially in hindsight, may be less than useful.  The classic example of this isf when a psychic predicts that a certain celebrity will experience a 'traumatic event' in the 'near future'.  The prediction is sufficiently vague that it can be 'validated' by any number of future events that may occur.

The when of a prediction is also an important part of specificity. A good prediction also specifies some kind of timeframe for a predicted event ot occur in.  The prediction that you will die is trivial in that barring amazing medical advances in the near future, it represents a certainty. Adding  detail such as 'dying in a traffic accident' or 'dying on February 22nd' adds specificity.  The principal value of specificity isn't that more detail is better in itself, but more detail can often make the forecast more useful.

Sometimes forecasts are made in the form of probabilities, such as  a '40% chance of rain tomorrow'.  This kind of forecast should not be viewed as a hedge, but as recognition of uncertainty, and an attempt at quantification by the forecaster.  In addition to signifying confidence, probability estimates in forecasts can also serve as key inputs to planning and decision-making processes. An example is if sales provides an estimate of 60% chance that a $100,000 sale will go through - one possible use for this type of input is in an expected value anaysis, where the total amount of the sale would be discounted to 60% of its nominal value ($60,000) for revenue planning purposes.  When relying on probability estimates, however, one should keep a sharp eye out towards how they were developed - I have worked with numerous sales people who regularly provide estimates of 110% that their predicted sales will occur :)


A Good Forecast Has the Possibility of Being Wrong

This may sound completely oxymoronic, as people generally find incorrect forecasts to be of little or no value, or sometimes even very costly. My reason for writing this is that a 'good' forecast may be viewed as a kind of scientific hypothesis.  Karl Popper, a philosopher of science, maintained that a critical part of a scientific theory is its ability to be falsified by contrary evidence.  For example, by Popper's criterion, the statement 'Space aliens may have visited the Earth' is not scientific while the opposing statement 'Space aliens have never visited the Earth' is. The first statement is not unscientific because of the opinion expressed, but because it is practically, and in principlel, impossible to disprove.  A lack of evidence of extraterrestrial landings just means that you have not been able to find any - but is not proof that these events have never happened. After all, the Earth is a big place and it has been around for four billion years - who is to say that ET didn't drop by 60 million years ago to check out the dinosaurs?

The latter statement, however,  can be readily disproven if someone can present credible evidence, such as a spacecraft or an actual alien.  This ability to prove the statement false, and the related ability to define what would consititute sufficient contrary evidence to do so   is what makes it 'scientific', not the particular opinion expressed.

Similarly, many predictions take the form of statements that can be shown to be either true or false after the predicted event either occurs or fails to occur.  For example, a prediction of which team will win a football game can be readily verified by looking at the game's final score.  Saying that the game could go 'one way or the other', isn't a forecast.

Some predictions, however, may not have binary outcomes, but numerical values or expected ranges - such as a weather forecast. In cases such as these, the forecast may include an expected range (5-8 inches of snow,  high temperature of 75-80F, etc..).  In cases like these, there are varying degrees of error. If the weatherman predicts a 8 inches  of snow, and you get 6 inches - that isn't too far off. If you get 80 degrees and sunny that is a lmuch larger degree of error.  


A Good Forecast is Made Through a Credible Process

GLENDOWER:   I can call spirits from the vasty deep.

HOTSPUR:  Why, so can I, or so can any man, But will they come when you do call for them?

Henry IV Part 1, Act 3, Scene 1, William Shakespeare

Much as anyone can 'call spirits', anybody can make predictions by any number of means.  While you are likely to consistently fail at spirit-calling, if you make enough forecasts, some of them are bound to turn out correct.  And sometimes, 'good' forecasters will be wrong.  How can we determine what a 'good' forecast is, especially if it for a one-time or rarely-ocurring event such as a presidential election?

A forecast can be compared to a decision in the sense that it is possible to separately consider the quality of the decision-making/forecasting process and the outcome/forecast.  A decision is considered to be 'good' if the process used to arrive at the decision is defensible.  For example, a medical diagnosis based on the results of generally accepted tests and interpretation of symptoms is an example of a 'good' decision. A medical diagnosis made on the basis of a coin flip (Heads you have cancer/tail you don't) would not.  Note that in using a coin-toss the doctor could come up with a correct diagnosis, but would have a difficult time justifying why anyone should believe it.

Although important, defensibility is only one criterion for determining if a forecasting process is any good. In cases where the forecast is made many times, the track record can be a far more important factor. This is especially applicable to statistical and machine-learning (AI) based forecasting models where there may be no rational justification for the model, but it produces accurate and consistent predictions. I would argue that when you have a history of previous forecasts long enough to evaulate statistically, the measured quality is far more important to determine method credibility (or lack of it) than logical arguments as to why the process 'should' work.

21 NOV 2016