v2.09.2007b

*manual not updated yet*

- Epoch size may be adjusted for the steepest-descend training algorithm - specified number of training vectors is randomly chosen at each iteration; for epoch size = 1 this is equivalent to on-line training (local approach).
- Training vectors are shuffled partially after each epoch.
- Weight variable may be specified for training vectors; this weight should be a non-negative number, vectors with lower weight are less important during the training.
- Support for SVM classification/regression added (uses SVM.NET v1.2 C# port of LIBSVM v2.6 code).
- Single filter condition changed to unlimited sequence of conditions encoded in C-like syntax, for example: (i1>2)&(i2<i3)&((o1>=0.5)|(t1==0.95)). This applies to: 1) Forwarding tabs in Transformation, Classification and Network blocks; 2) filter variable of different plots; 3) Save Data tab page of DataSet block. Project file format has changed - forwarding setup needs to be redone in old files.
- Many user interface changes: training/testing set selection moved to Connection Add dialog window; Transformation and Classification blocks have simplified and more uniform setup dialog windows; multiple training sets may be used with Classification block.

v29.06.2007e

- Neuron split added to the algorithm of the network growth (manual updated).
- Hold and Save phases training control options added.
- Fixed issue: filter variable of the Scatter Plot (2D) set to "nxx" had no effect.
- Improved detection of the training-stop conditions (for configurations with testing set connected during the training).
- Default values of the training algorithms changed.
- Small changes in the user interface.

v21.10.2006d

- Twin neurons measure modified - previous values should be divided by 2 to get similar results.
- Tuning of the neuron candidates initial weights.
- Fixed issue with log messages.
- Number of weights removed with OBS in one shot is now limited: total Lq < 5 Lq threshold and max 10 weights; then the network is retrained. This is due to possible large error jump in case of many weights with Lq << threshold.
- OBS code optimized (really small improvement).

v21.10.2006

- Changes in algorithm of adding new neurons: if new neuron was accepted and then killed, next new neuron candidates are not tried in the same layer. Initial weights (to the following layer) are randomized using the range corresponding to the existing connection weights (this makes "spikes" on the error(iterations) plot higher, don't worry about that).

v4.10.2006a

- ITanh error function added; use this function if you expect outliers or gross errors in the training data. I recommend to try this function especially if MSE was the best for your task until now.

v4.10.2006

- Algorithm for twin neurons joining improved (more pairs are detected); default neuron candidate pool size changed to 5 - pre-training of candidates is quite fast now, so this value seems to be reasonable; some other optimizations in the network structure reduction procedures.
- Optimal Brain Surgeon added (weight elimination technique). It may be applied at the end of the training to reduce the network size (sometimes whole neurons are removed), or it may support existing size adjustment algorithm when called iteratively before each attempt of removing redundant neurons. Be aware: this is time and memory consuming technique - O(N³ * M), where N is the number of the network coefficients and M is the training set size. Code will be optimized in further versions. Some examples and description in manual soon.
- SSE3 instructions (P4 Prescott and above) used, in a few places, not particularly faster, but why not to use them if they are here...
- Log window is now resizable and shows some more events; order of events is inverted (newest are on the top).

v25.09.2006

- Serious bug fixed in C/asm code (ANNLibs.dll library). This bug caused unstable training in some configurations of the hidden layer sizes. Library has been replaced also in archives with older versions of NetMaker.
- Log window added (menu View / Log Window). Systematically more events will be logged in further versions.
- Classification algorithms code optimized.

v19.09.2006

- Asymmetric error functions now can be inverted: err_new(x) = err_org(-x).

v7.09.2006c

- Shuffle button added to DataSet setup dialog window.
- Fixed bug in LBGBins classification.

v7.09.2006b

- Fixed bug in reloading DataSet when connected to Classification block as a source of training data.
- Optimization in user interface code.

v7.09.2006a

- Fixed issue with Network Error plot on a multithread and dual-core processors.
- One of the activation functions now has the proper name: Elliott function.

v7.09.2006

- New block type: Classification. Includes standard classification algorithm: kNN and bayesian estimator of conditional probability in two variations; description in manual soon.
- Retraining after pruning redundant and adding new neurons tuned.
- Fixed bug in JustRun method of Cascade-Correlation network (was there in two last versions).

v15.07.2006a

- Retraining after pruning optimized.

v15.07.2006

- Reduced number of retraining iterations after neuron pruning; this is an intermediate step to the final optimization of this part of the algorithm (previous versions performed full retraining, while only few iterations are required).
- Fixed bug in conjugated gradients algorithm (caused NaN exception).

v14.06.2006

- Final rearrangement of error functions e(d), where d = target - output; now we have:
    - old mean squared error (MSE) and fourth power of d (Pow4) that are suitable for all kind of output layers;
    - integrated hyperbolic arcus tangent of d (IAtanh1) for unipolar and centered sigmoid f_act of output layer;
    - rescaled integrated hyperbolic arcus tangent of d (IAtanh2) for bipolar output layers - those with f_actÎ(-1; 1);
    - (a∙d)² / (a∙d + 1), where a is a scaling factor: a = 0.75 (Asymm1, for unipolar output layers), a = 0.4 (Asymm2, for bipolar output layers);
    - e^-d + d - 1, AsymmL - asymmetric, with unlimited range of d and therefore suitable for all types of output layer.
- New element in neuron pruning algorithm: const neurons - description in manual soon.
- Some small rearrangements in menus, plot names.
- Fixed bug: XML network file used comma (,) instead of dot (.) as a decimal separator in a part of coefficients when system settings where set to use comma. Now dot is used no matter what is set in "regional settings" of Windows.

v21.05.2006

- One more layer type added: centered sigmoid f_actÎ(-0.5; 0.5).
- CCov error function changed to work with bipolar activation functions (and renamed to Asymm1).
- Another asymmetric error function added (Asymm2) - wide, suitable also for linear output layer.
- Delete button and tool-tips for all buttons added on project blocks.
- Blocks layout inside the project window is now independent from screen resolution. Your old project files need to be updated: 1) open *.NetPrj with a text editor; 2) divide each xpos, ypos, xsize, ysize entry by 10 (insert dot before last digit or just delete last digit) - this will bring old sizes and positions to new ranges (more or less...); save file; 3) open file with the new NetMaker and adjust positions and sizes as you like; save the project. Now, project window will look very similar when opened on another machine, with a different display dpi. Example files are updated already.

v3.05.2006c

- Fixed bug (removing dead neuron).

v3.05.2006b

- Bipolar arcus tangent activation function derivative corrected (was mixed with the unipolar version).

v3.05.2006a

- Added SSE optimizations for the new activation functions.

v3.05.2006

- New layer types (activation functions):
  - hyperbolic tangent (bipolar version of standard logistic-sigmoid function);
  - x/[2(1 + |x|)] + ½, unipolar, slowly approaching its asymptotes;
  - x/(1 + |x|), bipolar version of the above function;
  - arcus tangent, also slowly approaches asymptotes, but with a smooth derivative;
  - bipolar version of arcus tangent function.
- New Functions graph allows to plot the built in activation functions, theirs derivatives and error functions.
- User interface code recompiled - executable size significantly reduced, interface performance improved.

v20.04.2006

- AutoStop algorithm changed (s/m of the network weights watched instead of the weight differences in regular iteration intervals).

v4.04.2006

- Recurrent network model changed to RMLP. Recurrent network is now separate type from MLP network type (there is no longer recurrent input layer type visible to the user). Training algorithms have the same parameters as for MLP networks, plus two additional: Teacher Forcing factor and Forcing Decay factor - see manual for description.
- Project file format changed. Example files are updated and can be opened with new and old versions. If you have your own old files, comment lines that cause problems (with "//").
- Executable file size has been reduced due to code refactoring. NetMaker should be more stable also.

v11.03.2006

- Minor fix in the random number generator (bug was present in the last few versions, but it was not very significant to the networks).

v5.03.2006 (removed from server in strange circumstances...)

- Moved to Framework.NET 2.0 (with some optimizations using Generics).
- Histogram and XY Data Points plots automatically use names of chosen variables to label the axes.

v2.03.2006

- Last version running under Framework.NET 1.1.

Fixed bugs:
- Fixed problems with running under Windows 2000.
- Fixed issue with list of files and directories in Open/Save Dialog Windows.

v26.02.2006

- Naming of the variables added (DataSets with correctly assigned source files): click appropriate column header in the Setup/Preview to change the default name. Names are stored in the project file.

Bug discovered:
- Problems with running under Windows 2000.

v.22.11.2005

- XML network description may contain multiple transformations (Matrix and N(0,1)). Note: C++ code not updated yet.
- Genetic algorithm gap optimization removed.

Fixed bug:
- Corrected conversion between fixed structure and BP-trainable network.

v.28.08.2005a

- New file format for the network description (coefficients and N(0,1) transformation info in a single XML file). This file format is used by the new, pure C++ module that enables using the trained network in your programs. Code with simple example is available at main page.
- Linear correlation ration added to the legend of the XY Data Points scatter plot.

v.4.07.2005

ANN Engine:
- Networks with dynamic structure have the pool of candidate neurons that are going to be installed into the network structure. Pool value was fixed in previous version (3 for the Cascade-Correlation and 1 for the Back-Propagation networks).
- Look Back feature added for recurrent networks (analogous to Look Forward).
- Auto Stop algorithm improved.

Fixed bug:
- Transformation block set to QSVD caused hang on the next attempt to enter the Setup.

v.9.05.2005:

First version after a long break and many modifications.

Known problems:

• (solved) graphs: "Network Error" graph may not refresh from time to time - occurs especially while working with many networks running, just close the warning dialog window and it should go on without problems;
• (solved in v2.03.2006) open/save common dialog window: all the file/directory names disappear after changing the "File Type" on some systems (WinXP only?) - you must go one directory up and then return to previous directory to see them again (or push Open/Save button with empty File Name text box);

back to the main page