[R] What training algorithm does nnet package use?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[R] What training algorithm does nnet package use?

Wee-Jin Goh
Greetings list,

I've just swapped from the "neural" package to the "nnet" package and  
I've noticed that the training is orders of magnitude faster, and the  
results are way more accurate.

This leads me to wonder, what training algorithm is "nnet" using? Is  
it a modification on the standard backpropagation? Or a completely  
different algorithm? I'm trying to account for the speed differences  
between neural and nnet, and the documentation on the nnet package is  
rather sparse on what training algorithm is used (either that, or I'm  
getting blind and missed it totally).

Any help would be much appreciated.

Regards,
Wee-Jin

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] What training algorithm does nnet package use?

Wee-Jin Goh
Just to add to this, I also need to know what language is the "nnet"  
package written in? Is it in pure R or is it a wrapper for a C  
library. It really is performing very quickly, going through 200  
epochs in seconds when it took "neural" minutes, and neural is  
written in R.

It all may sound like a trivial question, but it is important as I  
need to know all this for the analysis I'm doing for a paper.

Cheers,
Wee-Jin


On 22 Nov 2006, at 15:41, Wee-Jin Goh wrote:

> Greetings list,
>
> I've just swapped from the "neural" package to the "nnet" package and
> I've noticed that the training is orders of magnitude faster, and the
> results are way more accurate.
>
> This leads me to wonder, what training algorithm is "nnet" using? Is
> it a modification on the standard backpropagation? Or a completely
> different algorithm? I'm trying to account for the speed differences
> between neural and nnet, and the documentation on the nnet package is
> rather sparse on what training algorithm is used (either that, or I'm
> getting blind and missed it totally).
>
> Any help would be much appreciated.
>
> Regards,
> Wee-Jin
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting- 
> guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] What training algorithm does nnet package use?

Dieter Menne
Wee-Jin Goh <wjgoh <at> brookes.ac.uk> writes:

>
> Just to add to this, I also need to know what language is the "nnet"  
> package written in? Is it in pure R or is it a wrapper for a C  
> library.

As usual, you can download the full source to find out what you want, but it's a
bit hidden. Simply said, nnet (R+C) is part of package MASS is part of bundle
VR, and can be downloaded as a tar.gz from

http://cran.at.r-project.org/src/contrib/Descriptions/VR.html

(No private flames, please, in case I should have mixed up package and bundle).

/*  nnet/nnet.c by W. N. Venables and B. D. Ripley  Copyright (C) 1992-2002
 *
 * weights are stored in order of their destination unit.
 * the array Conn gives the source unit for the weight (0 = bias unit)
 * the array Nconn gives the number of first weight connecting to each unit,
 * so the weights connecting to unit i are Nconn[i] ... Nconn[i+1] - 1.
 *
 */

#include <R.h>
#include <R_ext/Applic.h>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] What training algorithm does nnet package use?

Wee-Jin Goh
Hello,

Thanks for that. I've taken a look at the source code, and I see that  
the bulk of the processing is done in C, with R acting as a wrapper.  
Below is the function I think is doing the training in the network.

I'm guessing it's the standard Backpropagation with a decay term  
algorithm? Can anyone confirm if that's correct?

Cheers,
Wee-Jin

------------------------------------------------------------
void
VR_dfunc(double *p, double *df, double *fp)
{
     int   i, j;
     double sum1;

     for (i = 0; i < Nweights; i++)
        wts[i] = p[i];
     for (j = 0; j < Nweights; j++)
        Slopes[j] = 2 * Decay[j] * wts[j];
     TotalError = 0.0;
     for (i = 0; i < NTrain; i++) {
        for (j = 0; j < Noutputs; j++)
            toutputs[j] = TrainOut[i + NTrain * j];
        fpass(TrainIn + i, toutputs, Weights[i], NTrain);
        bpass(toutputs, Weights[i]);
     }
     sum1 = 0.0;
     for (i = 0; i < Nweights; i++)
        sum1 += Decay[i] * p[i] * p[i];
     *fp = TotalError + sum1;
     for (j = 0; j < Nweights; j++)
        df[j] = Slopes[j];
     Epoch++;
}
-----------------------------------------


On 23 Nov 2006, at 07:36, Dieter Menne wrote:

> Wee-Jin Goh <wjgoh <at> brookes.ac.uk> writes:
>
>>
>> Just to add to this, I also need to know what language is the "nnet"
>> package written in? Is it in pure R or is it a wrapper for a C
>> library.
>
> As usual, you can download the full source to find out what you  
> want, but it's a
> bit hidden. Simply said, nnet (R+C) is part of package MASS is part  
> of bundle
> VR, and can be downloaded as a tar.gz from
>
> http://cran.at.r-project.org/src/contrib/Descriptions/VR.html
>
> (No private flames, please, in case I should have mixed up package  
> and bundle).
>
> /*  nnet/nnet.c by W. N. Venables and B. D. Ripley  Copyright (C)  
> 1992-2002
>  *
>  * weights are stored in order of their destination unit.
>  * the array Conn gives the source unit for the weight (0 = bias unit)
>  * the array Nconn gives the number of first weight connecting to  
> each unit,
>  * so the weights connecting to unit i are Nconn[i] ... Nconn[i+1]  
> - 1.
>  *
>  */
>
> #include <R.h>
> #include <R_ext/Applic.h>
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting- 
> guide.html
> and provide commented, minimal, self-contained, reproducible code.


        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] What training algorithm does nnet package use?

Gavin Simpson
On Thu, 2006-11-23 at 09:39 +0000, Wee-Jin Goh wrote:

> Hello,
>
> Thanks for that. I've taken a look at the source code, and I see that  
> the bulk of the processing is done in C, with R acting as a wrapper.  
> Below is the function I think is doing the training in the network.
>
> I'm guessing it's the standard Backpropagation with a decay term  
> algorithm? Can anyone confirm if that's correct?
>
> Cheers,
> Wee-Jin

How about you take a look at Section 8.10 in MASS (the book; Venables &
Ripley (2002) Modern Applied Statistics in S 4th Edition, Springer.) and
Brian Ripley's 1996 book Pattern Recognition and Neural Networks,
Cambridge University Press.

Both of which are documented in the help page for nnet; see ?nnet. nnet
(the package) is support software for these books so you should consult
these references.

nnet is a package within the VR bundle by the way.

G

--
%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%
 Gavin Simpson                 [t] +44 (0)20 7679 0522
 ECRC & ENSIS, UCL Geography,  [f] +44 (0)20 7679 0565
 Pearson Building,             [e] gavin.simpsonATNOSPAMucl.ac.uk
 Gower Street, London          [w] http://www.ucl.ac.uk/~ucfagls/
 UK. WC1E 6BT.                 [w] http://www.freshwaters.org.uk
%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: [R] What training algorithm does nnet package use?

Hafsa Hassan
In reply to this post by Dieter Menne
Greetings list,

I am new to programming in R, and am using nnet() function for a project on neural networking.

Firslty i widh to ask if there is any pdf explaining the algorithm nnet uses, which could tell me what the objects of the nnet class, like 'conn', 'nconn, 'nsunits', n and 'nunits' mean, and how weights are calculated.
The package odf has little or no explanations, and the C +R surce code availabe at CRAN is too difficult to comprehend. Can anyone please help?

Also, i wish to know how the number of wieghts is calculated. when the nnet() command is run, it ouputs, on the console, the number of weights, and values of 'value'. But how do you calculate the bnumber of weights in nnet, say, if you are feeding it an MxN inputs dataframe (i.e. M observations, each having N inputs, like the iris dataset has M=150 and N=4), and getting, say, x number of outouts for each observation?

Thank you,
Hafsa