Download example (151kB) - project and data files.
Short introduction:
This is a popular "two-spiral" test in modified version. Two network models with dynamic structure are compared: Cascade-Correlation and MLP network with dynamic structure (my own design, trained with back-propagation algorithms - but don't worry, this is not a "my-algorithm-is-the-best" example). Like in previous examples, this is a separation of two classes, but this time the problem is much more complicated. Events from the Red and Blue classes are twisted in two interlocking spirals (used training set is shown in the image). Spirals are fuzzy and not clearly separable.
What is in the project:
Project contains three pairs of data-network blocks. Each data block is filled with the same training set of events (shown above). Networks are: MLP network with static structure, MLP network with dynamic structure and Cascade-Correlation network. Networks are ready to run the training, coefficients are randomized each time the project file is open. Training algorithm is set to quick-prop, parameters are set to reasonable values (identical for dynamic MLP and Cascade-Correlation, slightly different for static MLP). Archive contains also files which were used to obtain images presented below: trained network coefficients; testing set of two-spiral test events; set of events that uniformly cover 2D space of the network input. There is also training set used in the original, simple form of two-spiral test (plain_spiral.prn).
How to run this example:
Push the Go button of the Network block corresponding to the model you want to train; push there Go! button and wait for results. Save error info is turned on, you can observe the training error (select menu Edit - Add Graph - Network Error, right click in the plot window to open a context menu, select Add Net Source to open dialog window, select the network that is currently running and push OK). Multiple network may be trained at the same time (and the training progress may be plotted on a single image). Auto Stop is also turned on - when network is considered ready, training stops. Following image shows the training progress of the networks with dynamic structure:
Training process of Cascade-Correlation network consists of two interleaving parts - output layer training and new neuron candidates training (seen on the image as gaps in the blue plot). Usually dynamic MLP network (trained with back-propagation) reaches lower error values and finishes quicker than Cascade-Correlation network. This is a big disadvantage of Cascade-Correlation network - it doesn't work well if training set is "fuzzy". Further training of Cascade-Correlation network may lead to lower error value, but network completely loses its generalization capabilities. The default training sets may be replaced with events from the plain_spiral.prn file, where mentioned problem doesn't exist - then training of both models should lead to the very similar error values and the number of iterations also should be comparable.
When you already know how many hidden units are necessary to solve the fuzzy two-spiral problem you may try to prepare the network with a static structure. Try to start the training several times - network will stop in the local minima quite often. This is practically not observed in networks with dynamic structure. Oversizing the fixed structure usually helps and network learns much easier (2nd hidden layer is especially sensitive for two-spiral problem - MLP network with dynamic structure usually establishes no more than 7-8 hidden units in 2nd layer while at least 10 neurons is necessary for fixed structure to avoid the local minima problem). Single attempt of training the network with a fixed structure usually takes much less iterations than full training procedure of dynamic MLP network. Anyway, if the network training (with the fixed structure) stops too soon - try to turn off AutoStop and fix your desired number of iterations in MaxIter parameter. Training of the fixed structure may be too slow for auto-stop algorithm.
In the following images the network output is shown as a function of 2-dimensional network input. The same images may be obtained with 2D test data block - you should connect input of any network block to 2D test data block and run the network with JustRun method; then create XY Data Points graph and choose color palette instead of ConstColor (use default Z Axis Data variable which is o1).
As usual, network performance may be compared on a purity-efficiency plot. Results of both dynamic models are presented on such a plot below, together with the curve obtained for the MLP network trained with the static structure (file static_20-13.NetBin). Testing set of events (not shown to the network during the training) was used to calculate purities and efficiencies. The image shows that Cascade-Correlation network is a way worse than MLP models. Curves for dynamic and static MLP networks are very close to each other and for that reason the MSE error value calculated for testing events is also shown.
