632 estimator using boot package

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

632 estimator using boot package

Jin Minming
Dear All,

Anyone has some idea how to implement 632 estimator and leave-one out bootstraping method by using boot package.  I know the bootstrap package has this function, but it sounds not very flexible for my project.

Thanks,

Jim

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: 632 estimator using boot package

Frank Harrell
Bootstrapping does not leave one out.  As for .632 this is implemented in the rms package's validate and calibrate functions.  Note however that any claimed advantages of .632 over the ordinary optimism bootstrap seem to be a result only of the use of a discontinuous improper scoring role (proportion classified correctly).  The advantage may vanish when better scoring rules are used.  Simulations showing this may be found at http://biostat.mc.vanderbilt.edu/rms

Frank
Jin Minming wrote
Dear All,

Anyone has some idea how to implement 632 estimator and leave-one out bootstraping method by using boot package.  I know the bootstrap package has this function, but it sounds not very flexible for my project.

Thanks,

Jim

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Frank Harrell
Department of Biostatistics, Vanderbilt University
Reply | Threaded
Open this post in threaded view
|

Re: 632 estimator using boot package

Angelo Canty
In reply to this post by Jin Minming
There is an example of calculating the 0.632 prediction error estimator
in Chapter 6 of Davison & Hinkley (Practical 6.5)

I'm not sure what you mean by leave-one-out bootstrapping. If you
actually mean the jackknife then look at the empinf function. If you
mean subsampling, this can be implemented using type="parametric" and
supplying your own function which takes the data and returns a bootstrap
sample.

Angelo Canty

Jin Minming wrote:

> Dear All,
>
> Anyone has some idea how to implement 632 estimator and leave-one out bootstraping method by using boot package.  I know the bootstrap package has this function, but it sounds not very flexible for my project.
>
> Thanks,
>
> Jim
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>  


--
------------------------------------------------------------------
|   Angelo J. Canty                Email: [hidden email]     |
|   Mathematics and Statistics     Phone: (905) 525-9140 x 27079 |
|   McMaster University            Fax  : (905) 522-0935         |
|   1280 Main St. W.                                             |
|   Hamilton ON L8S 4K1                                          |

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: 632 estimator using boot package

Angelo Canty
Sorry, the final sentence should say sim="parametric"

Angelo Canty wrote:

> There is an example of calculating the 0.632 prediction error
> estimator in Chapter 6 of Davison & Hinkley (Practical 6.5)
>
> I'm not sure what you mean by leave-one-out bootstrapping. If you
> actually mean the jackknife then look at the empinf function. If you
> mean subsampling, this can be implemented using type="parametric" and
> supplying your own function which takes the data and returns a
> bootstrap sample.
>
> Angelo Canty
>
> Jin Minming wrote:
>> Dear All,
>>
>> Anyone has some idea how to implement 632 estimator and leave-one out
>> bootstraping method by using boot package.  I know the bootstrap
>> package has this function, but it sounds not very flexible for my
>> project.
>> Thanks,
>>
>> Jim
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>  
>
>


--
------------------------------------------------------------------
|   Angelo J. Canty                Email: [hidden email]     |
|   Mathematics and Statistics     Phone: (905) 525-9140 x 27079 |
|   McMaster University            Fax  : (905) 522-0935         |
|   1280 Main St. W.                                             |
|   Hamilton ON L8S 4K1                                          |

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: 632 estimator using boot package

Jin Minming
In reply to this post by Frank Harrell
Hello Frank,

Thanks for your advice.

I also used the validation using rms package. The results from rms indicate that the linear model used in my project is overfitted around 0.1 in terms of R2. But, the results from using ordinary boot method in boot package suggest the mean of R2 is even higher than the original one.  Then I guess maybe direclty using Boot package may not be a good idea for assessing the overfitting in the regression.

Jim


--- On Mon, 5/3/12, Frank Harrell <[hidden email]> wrote:

> From: Frank Harrell <[hidden email]>
> Subject: Re: [R] 632 estimator using boot package
> To: [hidden email]
> Date: Monday, 5 March, 2012, 17:06
> Bootstrapping does not leave one
> out.  As for .632 this is implemented in the
> rms package's validate and calibrate functions.  Note
> however that any
> claimed advantages of .632 over the ordinary optimism
> bootstrap seem to be a
> result only of the use of a discontinuous improper scoring
> role (proportion
> classified correctly).  The advantage may vanish when
> better scoring rules
> are used.  Simulations showing this may be found at
> http://biostat.mc.vanderbilt.edu/rms
>
> Frank
>
> Jin Minming wrote
> >
> > Dear All,
> >
> > Anyone has some idea how to implement 632 estimator and
> leave-one out
> > bootstraping method by using boot package.  I know
> the bootstrap package
> > has this function, but it sounds not very flexible for
> my project.
> >
> > Thanks,
> >
> > Jim
> >
> > ______________________________________________
> > R-help@ mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> > http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained,
> reproducible code.
> >
>
>
> -----
> Frank Harrell
> Department of Biostatistics, Vanderbilt University
> --
> View this message in context: http://r.789695.n4.nabble.com/632-estimator-using-boot-package-tp4446720p4446736.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> [hidden email]
> mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible
> code.
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: 632 estimator using boot package

Jin Minming
In reply to this post by Angelo Canty
Thanks a lot,
I will check that.

Jim

--- On Mon, 5/3/12, Angelo Canty <[hidden email]> wrote:

> From: Angelo Canty <[hidden email]>
> Subject: Re: [R] 632 estimator using boot package
> To: [hidden email]
> Date: Monday, 5 March, 2012, 18:19
> There is an example of calculating
> the 0.632 prediction error estimator in Chapter 6 of Davison
> & Hinkley (Practical 6.5)
>
> I'm not sure what you mean by leave-one-out bootstrapping.
> If you actually mean the jackknife then look at the empinf
> function. If you mean subsampling, this can be implemented
> using type="parametric" and supplying your own function
> which takes the data and returns a bootstrap sample.
>
> Angelo Canty
>
> Jin Minming wrote:
> > Dear All,
> >
> > Anyone has some idea how to implement 632 estimator and
> leave-one out bootstraping method by using boot
> package.  I know the bootstrap package has this
> function, but it sounds not very flexible for my project.
> > Thanks,
> >
> > Jim
> >
> > ______________________________________________
> > [hidden email]
> mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained,
> reproducible code.
> >   
>
>
> --
> ------------------------------------------------------------------
> |   Angelo J. Canty     
>           Email: [hidden email] 
>    |
> |   Mathematics and Statistics 
>    Phone: (905) 525-9140 x 27079 |
> |   McMaster University     
>       Fax  : (905) 522-0935   
>      |
> |   1280 Main St. W.     
>                
>                
>        |
> |   Hamilton ON L8S 4K1     
>                
>                
>     |
>
> ______________________________________________
> [hidden email]
> mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible
> code.
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: 632 estimator using boot package

Frank Harrell
In reply to this post by Jin Minming
Jim if I understand you, you are using the ordinary bootstrap instead of the Efron-Gong "optimism bootstrap".  I don't think the ordinary bootstrap is applicable here.
Frank
Jin Minming wrote
Hello Frank,

Thanks for your advice.

I also used the validation using rms package. The results from rms indicate that the linear model used in my project is overfitted around 0.1 in terms of R2. But, the results from using ordinary boot method in boot package suggest the mean of R2 is even higher than the original one.  Then I guess maybe direclty using Boot package may not be a good idea for assessing the overfitting in the regression.

Jim


--- On Mon, 5/3/12, Frank Harrell <[hidden email]> wrote:

> From: Frank Harrell <[hidden email]>
> Subject: Re: [R] 632 estimator using boot package
> To: [hidden email]
> Date: Monday, 5 March, 2012, 17:06
> Bootstrapping does not leave one
> out.  As for .632 this is implemented in the
> rms package's validate and calibrate functions.  Note
> however that any
> claimed advantages of .632 over the ordinary optimism
> bootstrap seem to be a
> result only of the use of a discontinuous improper scoring
> role (proportion
> classified correctly).  The advantage may vanish when
> better scoring rules
> are used.  Simulations showing this may be found at
> http://biostat.mc.vanderbilt.edu/rms
>
> Frank
>
> Jin Minming wrote
> >
> > Dear All,
> >
> > Anyone has some idea how to implement 632 estimator and
> leave-one out
> > bootstraping method by using boot package.  I know
> the bootstrap package
> > has this function, but it sounds not very flexible for
> my project.
> >
> > Thanks,
> >
> > Jim
> >
> > ______________________________________________
> > R-help@ mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> > http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained,
> reproducible code.
> >
>
>
> -----
> Frank Harrell
> Department of Biostatistics, Vanderbilt University
> --
> View this message in context: http://r.789695.n4.nabble.com/632-estimator-using-boot-package-tp4446720p4446736.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> [hidden email]
> mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible
> code.
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Frank Harrell
Department of Biostatistics, Vanderbilt University
Reply | Threaded
Open this post in threaded view
|

Re: 632 estimator using boot package

Jin Minming
Thanks a lot. I have changed the calculation method by using optimism defined by Efron. The results from using boot and rms packages are quite close now.

Jim


--- On Tue, 6/3/12, Frank Harrell <[hidden email]> wrote:

> From: Frank Harrell <[hidden email]>
> Subject: Re: [R] 632 estimator using boot package
> To: [hidden email]
> Date: Tuesday, 6 March, 2012, 3:23
> Jim if I understand you, you are
> using the ordinary bootstrap instead of the
> Efron-Gong "optimism bootstrap".  I don't think the
> ordinary bootstrap is
> applicable here.
> Frank
>
> Jin Minming wrote
> >
> > Hello Frank,
> >
> > Thanks for your advice.
> >
> > I also used the validation using rms package. The
> results from rms
> > indicate that the linear model used in my project is
> overfitted around 0.1
> > in terms of R2. But, the results from using ordinary
> boot method in boot
> > package suggest the mean of R2 is even higher than the
> original one.  Then
> > I guess maybe direclty using Boot package may not be a
> good idea for
> > assessing the overfitting in the regression.
> >
> > Jim
> >
> >
> > --- On Mon, 5/3/12, Frank Harrell
> &lt;f.harrell@&gt; wrote:
> >
> >> From: Frank Harrell &lt;f.harrell@&gt;
> >> Subject: Re: [R] 632 estimator using boot package
> >> To: r-help@
> >> Date: Monday, 5 March, 2012, 17:06
> >> Bootstrapping does not leave one
> >> out.  As for .632 this is implemented in the
> >> rms package's validate and calibrate functions. 
> Note
> >> however that any
> >> claimed advantages of .632 over the ordinary
> optimism
> >> bootstrap seem to be a
> >> result only of the use of a discontinuous improper
> scoring
> >> role (proportion
> >> classified correctly).  The advantage may vanish
> when
> >> better scoring rules
> >> are used.  Simulations showing this may be found
> at
> >> http://biostat.mc.vanderbilt.edu/rms
> >>
> >> Frank
> >>
> >> Jin Minming wrote
> >> >
> >> > Dear All,
> >> >
> >> > Anyone has some idea how to implement 632
> estimator and
> >> leave-one out
> >> > bootstraping method by using boot package.  I
> know
> >> the bootstrap package
> >> > has this function, but it sounds not very
> flexible for
> >> my project.
> >> >
> >> > Thanks,
> >> >
> >> > Jim
> >> >
> >> >
> ______________________________________________
> >> > R-help@ mailing list
> >> > https://stat.ethz.ch/mailman/listinfo/r-help
> >> > PLEASE do read the posting guide
> >> > http://www.R-project.org/posting-guide.html
> >> > and provide commented, minimal,
> self-contained,
> >> reproducible code.
> >> >
> >>
> >>
> >> -----
> >> Frank Harrell
> >> Department of Biostatistics, Vanderbilt University
> >> --
> >> View this message in context:
> >> http://r.789695.n4.nabble.com/632-estimator-using-boot-package-tp4446720p4446736.html
> >> Sent from the R help mailing list archive at
> Nabble.com.
> >>
> >> ______________________________________________
> >> R-help@
> >> mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-help
> >> PLEASE do read the posting guide
> >> http://www.R-project.org/posting-guide.html
> >> and provide commented, minimal, self-contained,
> reproducible
> >> code.
> >>
> >
> > ______________________________________________
> > R-help@ mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> > http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained,
> reproducible code.
> >
>
>
> -----
> Frank Harrell
> Department of Biostatistics, Vanderbilt University
> --
> View this message in context: http://r.789695.n4.nabble.com/632-estimator-using-boot-package-tp4446720p4448633.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> [hidden email]
> mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible
> code.
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.