|
Network
structure setup (Setup
button)
Setup has two
independent controls:
Name - Name of
the block. Should not contain white spaces nor "\"
symbols - these characters may lead to mistakes in
some cases.
Net Type - Neural network type:
BP Training is a multi-layer perceptron
network with a supervised, back-propagation
training pattern. Now it is implemented in a
feed-forward (MLP) and recursive (RMLP,
back-propagation through time training method)
versions, both heavily optimized for speed (SSE
is used if detected). Also, wide range of the
training algorithms is available for this network
types.
Cascade-Correlation
is an architecture and a supervised training
pattern proposed by
S. Fahlman in 1991. It is an
efficient model and it learns quickly, but it
also has some disadvantages (see
Back-Propagation vs Cascade-Correlation example).
Other controls are grouped in tabs: manual
structure modifications, file operations and
training/testing set selection.
- manual
modifications
Network structure may be fixed by user or it can be
adjusted automatically during the training, but
even in this case initial setup of some
parameters is required. Number of network inputs
and output neurons is determined by particular
network application. Hidden neuron numbers and
activation (transfer) functions (fact)
should be specified by user. All neurons in the
layer use the same activation function, but
fact
may differ between layers. In some cases
fact
of output layer is determined by an application
- like linear function in estimation tasks, if
output value range is unlimited. More on
available activation functions
here.
inps: Number of the network inputs (input
vector length) and input layer type. Type is set
to
Input
for feed-forward MLP networks and it is set to
Recurent,
for RMLP networks.
hid0/1: Number
of the hidden neurons and the hidden layer's
activation function type. Note, that the network
may consist of a different types of layers (and
it is recommended solution in some cases),
however some combinations may be impossible to
train.
outs:
Number of the network outputs (output vector
length) and output layer activation function
type (also may differ from the activation
function of the hidden layers).
randomize coeffs: If checked, neuron
interconnection coefficients will be randomized
after dialog window is closed even if there were
no changes to the network model.
Look
Ahead/Back: number of events that
follow/precede current event and are shown to
the network with current event.
Controls
enabled only if
RMLP
network type is selected:
Rank
- number of the previous network outputs that
are shown to the network with the current event;
clear state - if checked, feedback loop
is reset at the end of processing each
DataSet.
- reading and
writing network to a file
Reading / writing is very simple - just put the
name of the file in the text box or select it
through a common
Open
/
Save
dialog window (push "..." button). Then push
Load /
Save
button and it's done. Default type of the file
is
Binary,
it can be switched to
ASCII
or
XML
(XML
format is used by C++ module) in the dialog
window.
Preprocessing info may be included in
XML
files - select
preprocessing
check box and add the source
Transformation
blocks through the dialog window (click the
List
button).
--top--
--up-- |
|
Training parameters setup, run options (Go
button)
- training algorithms and
parameters
Training Method:
List of available optimization algorithms. More
on algorithms and parameters
here.
JustRun:
This special entry that appears for every type of
network. It stands for running the network
over the input DataSets without any training
operations, just to calculate the network
outputs.
Method Params: List of parameters
available for the selected training method. Only
common parameters are explained below. More on
algorithms and parameters
here.
AutoStop:
If off (=0), training stops at the first
reasonable iteration after
MaxIter
iterations (not when the error still
decreases rapidly or when the algorithm is
making an attempt that just supports main
stream of
calculations). If on (=1), a set of
conditions (stable error, stable neuron
interconnections...) is checked every
MaxIter
iterations and the training is stopped basing on
the result of this checks. Built-in stop
conditions are quite sophisticated, with
varying priorities for different states of
the training and they cannot be
modified by the user. Sensitivity of
the auto-stop algorithm can by modified
indirectly by
MaxIter
parameter. If training stops too early, use
bigger value of
MaxIter.
MaxIter: Number of the training
iterations / stop conditions checking
interval.
MinError: Satisfying
error level. If reached, training stops.
Rarely used and usually set to its default
value (10-6).
RMPL networks:
TeacherForcing:
mixing factor
f
for the recurrent input calculation
inp = out(t-n)∙(1-f)
+ tgt(t-n)∙f,
where
out is the network output and
tgt is
target value; if
f
= 1,
recurrent input is replaced by the desired
target value completely, if
f
= 0
then the teacher
forcing is disabled.
ForcingDecay:
Decay factor
d
for the teacher forcing factor; if the network error
decreases,
f
is recalculated at each iteration as
f(n)
=
d∙f(n-1);
if
f
= 1
then the
teacher forcing factor is constant,
f
= 0
is forbidden.
Dynamic Structure: Enabled only for
Back-Propagation networks. Controls automatic
adjustments of the hidden neurons number.
Add new
neurons: enables growth of the network
structure;
pool: number of
random candidates - each time the network size
increases the pool of new neurons is trained
and the best one is selected to extend the
network (...but only if it seems to be
useful to the whole network);
neuron
split: enables splitting of the hidden
neuron with the highest variation of the
output error (d);
pair of new neurons is then pre-trained;
neurons are built into the network if they
give better results than random candidates.
Remove neurons:
enables pruning of the unused or redundant
neurons;
twin
- measure of the difference between
activations of two neurons that will be
replaced with one equivalent neuron;
dead
- measure of significance of the neurons
that may be safely removed;
const
- measure of neuron activation
s/m
that will be used to consider neuron as
constant (modification of bias in following
layer is done when this type of neuron is
removed); increasing these
values results in more removed neurons.
OBS (Optimal Brain Surgeon):
technique of eliminating single connections
from the network; if
OBS
is enabled, algorithm is applied at the end
of training process; if
iterative
is also checked, algorithm is applied before
each attempt of neurons removing (be aware,
this is time and memory consuming technique
-
O(N3∙M),
where
N
is the number of the network coefficients
and
M
is the training set size);
Lq
is the measure of allowed error increase
caused by removing the connection; you can
read more about the algorithm
here (this is not a simple cutting of
connections).
Control Thread:
Save error
info: enables storing information about
the network training/testing error in each
iteration. Required for
Network Error
plot.
Save time info: enables
measuring of the time of each iteration.
Progress window: enables control
window with training statistics.
Hold: if checked, training will pause
before and after each structure
modification.
Save phases: if
checked, network model will be stored in
files before and after each structure
modification.
Shuffle data:
enables randomized reordering of the
training vectors during the training.
- forward mapping and filter
When network finishes processing the input
DataSets,
theirs contents is sent to the output
DataSets
according to the forward mapping specifications.
Destination events in each output
DataSet
may be composed in a different way. Select a
DataSet
in the
Output Data Set
list and put desired
formatting
strings
for destination event vectors. In the image
above there is a simple example:
-
destination event input vector will be composed
of the first and second element of the source
event input vector;
- destination event
output vector will be created with the length=1
and it will contain first element of the network
answer vector (output vector);
- destination
event target and not-used vector will not be
allocated.
Events stored in the output
DataSets
can be filtered (in the example from the image above
only events with the t1=0.95 will be sent to the
"signal"
DataSet).
- pre- and post-run commands
Pre-run commands (Initialization
list) are executed before starting the network
calculations. Post-run commands (Postprocessing
list) are executed when network finishes
calculations and sends data to the output sets.
Available commands:
Randomize()
- randomizes neuron interconnection weights.
ReadCoeffs(filename) - reads the
network structure from a file.
SaveCoeffs([filename[,ascii|bin[,y|n]]])
- writes down the network structure to a
file; if no
filename
is specified the name from
Setup
/
Save
tab is used, if
filename
in not specified in
Setup
the block name is used;
ascii/bin
option selects the file type (default valule
is also taken from
Setup
/
Save
tab);
y
option forces overwriting the existing file,
default is
n.
SaveData(dataset,[filename[,ascii|bin]])
- saves the contents of the
DataSet
connected to the network (input or output);
dataset
is the name of the
DataSet
block;
filename
is the name of the destination file, if not
specified the name from
Setup
/
Save
Data
tab is used, if
filename
in not specified in
Setup
the block name is used;
ascii/bin
option selects the file type (default value
is also taken from
Setup
/
Save
tab); format of data written to the file is
the same as specified in
Setup
/
Save
Data
tab.
RemoveSampleUpTo(dataset,nlast) -
removes all events from the
dataset
except
nlast
events.
ClearData(dataset1[,dataset2,...]) -
removes all events from specified list of
DataSets.
TriggChainBreak(counter) - Stops
triggering connected
Transform
/
Network
blocks after counter triggers. Useful for
creating finite loops in time-sequence
prediction.
ClosePrj() - stops
all activity and closes current project.
CloseApp() - stops all activity and
closes
NetMaker
instance.
- triggering
If
Manual
mode is selected,
Go!
button is enabled and it releases
calculations. If
Internal
mode is chosen -
Source
button is enabled and it allows to select source
of the trigger through the common
Connection Add
dialog window (Go!
button is disabled); processing starts when one
of the selected sources finished its own task.
--top--
--up-- |
Input list ("I/O"
button)
"I/O"
button opens the list of the training and
testing
DataSets
(a bit extended
Connection Add
dialog window). Double-click on the item from the
Available
list adds new training set. Use lower "<<"
button to add new item to the
Testing sets
list. Double-click on the item
from the
Training Sets
and
Testing Sets
lists removes connection.
When both training and testing sets are present
during the training process, training sets are
used to calculate neuron interconnection weights
changes, then the error on the testing sets is
calculated. If the error on the testing sets
seems to increase constantly, training is
stopped. This is a good way to avoid
overtraining.
--top--
--up-- |
Output list (">>"
button)
Connects the output of the
Network
block to the destination
DataSets.
Opens common
Connection Add
dialog window.
--top--
--up-- |
|
|
