What is the problem of induction, and why does it matter for forecasting?

The problem of induction asks how we can rationally justify predictions about unobserved things based on past observations. If bread of a certain appearance has nourished us so far, we expect the next piece to do so too, and we also tend to believe such bread is nourishing in general. These steps from the observed to the unobserved are called inductive inferences. The philosophical difficulty is that it is hard to say what reasoning, if any, supports them. Forecasting is built on this kind of inference, so the problem bears directly on it. [1] [2]

The problem comes from David Hume, in A Treatise of Human Nature (1739) and, in shorter form, An Enquiry concerning Human Understanding (1748). Hume held that there are only two kinds of argument, demonstrative and probable. A demonstrative argument would have to show that the opposite conclusion is a contradiction, but it is not contradictory to suppose that the future will differ from the past. A probable argument for the assumption that nature is uniform would itself presuppose that uniformity, and so would be circular. [1]

Hume did not dispute that people draw such inferences. His question was about their foundation, meaning the process of argument they rest on. The Stanford Encyclopedia summarizes his conclusion as being that our tendency to project past regularities into the future is not underpinned by reason. [1]

Philosophers disagree about the significance of the problem. Some have tried to solve it, and a significant number have accepted Hume's conclusion that it cannot be solved. Others argue that his argument does not establish a far-reaching skepticism, either because he never intended it to or because the argument is misformulated. Still others regard it as one of the most profound challenges possible, since it questions a basic way we form knowledge. Bertrand Russell said that if the problem cannot be solved, there is no intellectual difference between sanity and insanity. [1]

Justifying induction by its past success is circular, and the same trick can be used to pseudo-justify counter-induction, which predicts the opposite. Bayesian methods do not escape the issue: a prior probability distribution also needs justifying, and the reply that evidence will wash out the prior has limits. Some priors, such as Carnap's uniform measure over all possible event-sequences, make learning from experience impossible, and for any finite amount of evidence a suitably biased prior can prevent learning from it. [3]

Reichenbach argued that induction is at least the best available method, an optimality justification rather than a guarantee of reliability. Critics note that methods superior to ordinary induction, such as clairvoyance, cannot be excluded in advance. Gerhard Schurz proposes meta-induction, which applies induction to prediction methods by weighing their track records, for example by imitating the best method so far. He argues this is optimal a priori and gives an indirect, non-circular justification for ordinary induction to the extent that it outperforms the alternatives. [3] [4]

Any forecast that projects observed patterns forward relies on induction, so no forecasting method can be proven reliable from past performance alone. Meta-induction researchers suggest a practical consequence: because different methods perform best in different environments, especially when the environment changes unpredictably, aggregating the predictions of several methods according to their track records can improve forecasting. The problem has also been discussed in relation to machine learning, where generalizing from data raises the same question. [3] [5]