Before attempting an ANN model on a problem, one should if possible estimate
the outcome. For a classification problem, the optimal classification
performance is the Bayes limit or Bayes error [34],
which equals the Bayes risk with zero-one loss function. This
upper classification limit can be estimated by the use of simpler
classifiers, like the k-nearest-neighbours or Parzen windows
[34,50,62]
. With such an estimate at hand, it is much
easier to evaluate the quality of the ANN model.
To determine the termination point for the training it is customary to use a validation data set. This validation set is not used directly in the training, i.e. not presented to the network, but used indirectly to monitor the performance on unknown data. A deteriorating performance on the validation set signals that the ANN is overlearning the training data and that training should be stopped. When the training is stopped, a test set can be used to estimate the generalization performance. It is however imperative that the validation data are not used in the test set, since it is indirectly used in the training to choose a stopping point.
In cases where data is scarce and the use of a validation set is too costly, one can instead use a threshold value on the training error. For instance, when computing CV estimates one can train each network until it reaches a prespecified training error, which has been determined by a couple of trial runs.