COMPUTING WITH FINITE AND INFINITE NETWORKS
Ole Winther
Using statistical mechanics results, I calculate learning curves
(average generalization error) for Gaussian processes (GPs) and Bayesian
neural networks (NNs) used for regression. Applying the results to
learning a teacher defined by a two-layer network, I can directly
compare GP and Bayesian NN learning. I find that a GP in general
requires O(d^s)-training examples to learn input features of order s (d
is the input dimension), whereas a NN can learn the task with order the
number of adjustable weights training examples. Since a GP can be
considered as an infinite NN, the results show that even in the Bayesian
approach,
it is important to limit the complexity of the learning machine. The
theoretical findings are confirmed in simulations with analytical GP
learning and a NN mean field algorithm.
LUTP 00-20