Re: How to compare areas under ROC curves calculated with ROC R package

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: How to compare areas under ROC curves calculated with ROC R package

Tuszynski, Jaroslaw W.
I might be missing something but I thought that AUC was a measure for
comparing ROC curves, so there is nothing else needed to "compare" them. The
larger AUC is the higher correlation of 2 variables compared. No other
measures or calculations are needed.

Jarek Tuszynski

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Laurent Fanchon
Sent: Wednesday, March 15, 2006 11:57 AM
To: [hidden email]
Subject: [R] How to compare areas under ROC curves calculated with ROCR
package

Dear all,

I try to compare the performances of several parameters to diagnose
lameness in dogs.
I have several ROC curves from the same dataset.
I plotted the ROC curves and calculated AUC with the ROCR package.

I would like to compare the AUC.
I used the following program I found on R-help archives :
 
From: Bernardo Rangel Tura
Date: Thu 16 Dec 2004 - 07:30:37 EST

seROC<-function(AUC,na,nn){
a<-AUC
q1<-a/(2-a)
q2<-(2*a^2)/(1+a)
se<-sqrt((a*(1-a)+(na-1)*(q1-a^2)+(nn-1)*(q2-a^2))/(nn*na))
se
}

cROC<-function(AUC1,na1,nn1,AUC2,na2,nn2,r){
se1<-seROC(AUC1,na1,nn1)
se2<-seROC(AUC2,na2,nn2)

sed<-sqrt(se1^2+se2^2-2*r*se1*se2)
zad<-(AUC1-AUC2)/sed
p<-dnorm(zad)
a<-list(zad,p)
a
}

The author of this script says: "The first function (seROC) calculate the
standard error of ROC curve, the
second function (cROC) compare ROC curves."

What do you think of this script?
Is there any function to do it better in ROCR?

Any help would be greatly appreciated.

Laurent Fanchon
DVM, MS
Ecole Nationale Vétérinaire d'Alfort
FRANCE

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Reply | Threaded
Open this post in threaded view
|

Re: How to compare areas under ROC curves calculated with ROC R package

Frank Harrell
Tuszynski, Jaroslaw W. wrote:
> I might be missing something but I thought that AUC was a measure for
> comparing ROC curves, so there is nothing else needed to "compare" them. The
> larger AUC is the higher correlation of 2 variables compared. No other
> measures or calculations are needed.
>
> Jarek Tuszynski

A single AUC does not compare two competing predictors; it only measures
the predictive discrimination of a single predictor (or a single model).

Note that testing for differences in ROC areas results in a test with
low power.  There are much more powerful approaches such as likelihood
ratio chi-square tests for embedded models.

The code you included below seems to make some strong assumptions about
how AUC is estimated.  Nonparametric AUC estimators (related to the
Wilcoxon test, for example, which is a preferred method) do not have
simple formulas for standard errors.

Frank Harrell

>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of Laurent Fanchon
> Sent: Wednesday, March 15, 2006 11:57 AM
> To: [hidden email]
> Subject: [R] How to compare areas under ROC curves calculated with ROCR
> package
>
> Dear all,
>
> I try to compare the performances of several parameters to diagnose
> lameness in dogs.
> I have several ROC curves from the same dataset.
> I plotted the ROC curves and calculated AUC with the ROCR package.
>
> I would like to compare the AUC.
> I used the following program I found on R-help archives :
>  
> From: Bernardo Rangel Tura
> Date: Thu 16 Dec 2004 - 07:30:37 EST
>
> seROC<-function(AUC,na,nn){
> a<-AUC
> q1<-a/(2-a)
> q2<-(2*a^2)/(1+a)
> se<-sqrt((a*(1-a)+(na-1)*(q1-a^2)+(nn-1)*(q2-a^2))/(nn*na))
> se
> }
>
> cROC<-function(AUC1,na1,nn1,AUC2,na2,nn2,r){
> se1<-seROC(AUC1,na1,nn1)
> se2<-seROC(AUC2,na2,nn2)
>
> sed<-sqrt(se1^2+se2^2-2*r*se1*se2)
> zad<-(AUC1-AUC2)/sed
> p<-dnorm(zad)
> a<-list(zad,p)
> a
> }
>
> The author of this script says: "The first function (seROC) calculate the
> standard error of ROC curve, the
> second function (cROC) compare ROC curves."
>
> What do you think of this script?
> Is there any function to do it better in ROCR?
>
> Any help would be greatly appreciated.
>
> Laurent Fanchon
> DVM, MS
> Ecole Nationale Vétérinaire d'Alfort
> FRANCE
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
>


--
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Frank Harrell
Department of Biostatistics, Vanderbilt University