about a p-value < 2.2e-16

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

about a p-value < 2.2e-16

Bogdan Tanasa
 <https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16>
Dear all,

i would appreciate having your advice on the following please :

in R, the wilcox.test() provides "a p-value < 2.2e-16", when we compare
sets of 1000 genes expression (in the genomics field).

however, the journal asks us to provide the exact p value ...

would it be legitimate to write : "p-value = 0" ? thanks a lot,

-- bogdan

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Spencer Graves-4
       I would push back on that from two perspectives:


             1.  I would study exactly what the journal said very
carefully.  If they mandated "wilcox.test", that function has an
argument called "exact".  If that's what they are asking, then using
that argument gives the exact p-value, e.g.:


 > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)

         Wilcoxon rank sum exact test

data:  rnorm(100) and rnorm(100, 2)
W = 691, p-value < 2.2e-16


             2.  If that's NOT what they are asking, then I'm not
convinced what they are asking makes sense:  There is is no such thing
as an "exact p value" except to the extent that certain assumptions
hold, and all models are wrong (but some are useful), as George Box
famously said years ago.[1]  Truth only exists in mathematics, and
that's because it's a fiction to start with ;-)


       Hope this helps.
       Spencer Graves


[1]
https://en.wikipedia.org/wiki/All_models_are_wrong


On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:

>   <https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16>
> Dear all,
>
> i would appreciate having your advice on the following please :
>
> in R, the wilcox.test() provides "a p-value < 2.2e-16", when we compare
> sets of 1000 genes expression (in the genomics field).
>
> however, the journal asks us to provide the exact p value ...
>
> would it be legitimate to write : "p-value = 0" ? thanks a lot,
>
> -- bogdan
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

plangfelder
I thinnk the answer is much simpler. The print method for hypothesis
tests (class htest) truncates the p-values. In the above example,
instead of using

wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)

and copying the output, just print the p-value:

tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
tst$p.value

[1] 2.988368e-32


I think this value is what the journal asks for.

HTH,

Peter

On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves
<[hidden email]> wrote:

>
>        I would push back on that from two perspectives:
>
>
>              1.  I would study exactly what the journal said very
> carefully.  If they mandated "wilcox.test", that function has an
> argument called "exact".  If that's what they are asking, then using
> that argument gives the exact p-value, e.g.:
>
>
>  > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>
>          Wilcoxon rank sum exact test
>
> data:  rnorm(100) and rnorm(100, 2)
> W = 691, p-value < 2.2e-16
>
>
>              2.  If that's NOT what they are asking, then I'm not
> convinced what they are asking makes sense:  There is is no such thing
> as an "exact p value" except to the extent that certain assumptions
> hold, and all models are wrong (but some are useful), as George Box
> famously said years ago.[1]  Truth only exists in mathematics, and
> that's because it's a fiction to start with ;-)
>
>
>        Hope this helps.
>        Spencer Graves
>
>
> [1]
> https://en.wikipedia.org/wiki/All_models_are_wrong
>
>
> On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
> >   <https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16>
> > Dear all,
> >
> > i would appreciate having your advice on the following please :
> >
> > in R, the wilcox.test() provides "a p-value < 2.2e-16", when we compare
> > sets of 1000 genes expression (in the genomics field).
> >
> > however, the journal asks us to provide the exact p value ...
> >
> > would it be legitimate to write : "p-value = 0" ? thanks a lot,
> >
> > -- bogdan
> >
> >       [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Bogdan Tanasa
In reply to this post by Spencer Graves-4
Dear Spencer, thank you very much for your prompt email and help. When
using :

> wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
W = 698, p-value < 2.2e-16

> wilcox.test(rnorm(100), rnorm(100, 2), exact=FALSE)
W = 1443, p-value < 2.2e-16

and in both cases p-value < 2.2e-16. By "exact" p-value, i have meant the
"precise" p-value ;

If I may ask please, could we write p-value = 0 ?

i have noted a similar conversation on stackexchange, although the answer
is not very clear (to me).

https://stats.stackexchange.com/questions/78839/how-should-tiny-p-values-be-reported-and-why-does-r-put-a-minimum-on-2-22e-1

thanks again,

bogdan

On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves <
[hidden email]> wrote:

>        I would push back on that from two perspectives:
>
>
>              1.  I would study exactly what the journal said very
> carefully.  If they mandated "wilcox.test", that function has an
> argument called "exact".  If that's what they are asking, then using
> that argument gives the exact p-value, e.g.:
>
>
>  > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>
>          Wilcoxon rank sum exact test
>
> data:  rnorm(100) and rnorm(100, 2)
> W = 691, p-value < 2.2e-16
>
>
>              2.  If that's NOT what they are asking, then I'm not
> convinced what they are asking makes sense:  There is is no such thing
> as an "exact p value" except to the extent that certain assumptions
> hold, and all models are wrong (but some are useful), as George Box
> famously said years ago.[1]  Truth only exists in mathematics, and
> that's because it's a fiction to start with ;-)
>
>
>        Hope this helps.
>        Spencer Graves
>
>
> [1]
> https://en.wikipedia.org/wiki/All_models_are_wrong
>
>
> On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
> >   <
> https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16>
> > Dear all,
> >
> > i would appreciate having your advice on the following please :
> >
> > in R, the wilcox.test() provides "a p-value < 2.2e-16", when we compare
> > sets of 1000 genes expression (in the genomics field).
> >
> > however, the journal asks us to provide the exact p value ...
> >
> > would it be legitimate to write : "p-value = 0" ? thanks a lot,
> >
> > -- bogdan
> >
> >       [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Bogdan Tanasa
In reply to this post by plangfelder
Dear Peter, thanks a lot. yes, we can see a very precise p-value, and that
was the request from the journal.

if I may ask another question please : what is the meaning of "exact=TRUE"
or "exact=FALSE" in wilcox.test ?

i can see that the "numerically precise" p-values are different. thanks a
lot !

tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
tst$p.value
[1] 8.535524e-25

tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=FALSE)
tst$p.value
[1] 3.448211e-25

On Thu, Mar 18, 2021 at 10:15 PM Peter Langfelder <
[hidden email]> wrote:

> I thinnk the answer is much simpler. The print method for hypothesis
> tests (class htest) truncates the p-values. In the above example,
> instead of using
>
> wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>
> and copying the output, just print the p-value:
>
> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> tst$p.value
>
> [1] 2.988368e-32
>
>
> I think this value is what the journal asks for.
>
> HTH,
>
> Peter
>
> On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves
> <[hidden email]> wrote:
> >
> >        I would push back on that from two perspectives:
> >
> >
> >              1.  I would study exactly what the journal said very
> > carefully.  If they mandated "wilcox.test", that function has an
> > argument called "exact".  If that's what they are asking, then using
> > that argument gives the exact p-value, e.g.:
> >
> >
> >  > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> >
> >          Wilcoxon rank sum exact test
> >
> > data:  rnorm(100) and rnorm(100, 2)
> > W = 691, p-value < 2.2e-16
> >
> >
> >              2.  If that's NOT what they are asking, then I'm not
> > convinced what they are asking makes sense:  There is is no such thing
> > as an "exact p value" except to the extent that certain assumptions
> > hold, and all models are wrong (but some are useful), as George Box
> > famously said years ago.[1]  Truth only exists in mathematics, and
> > that's because it's a fiction to start with ;-)
> >
> >
> >        Hope this helps.
> >        Spencer Graves
> >
> >
> > [1]
> > https://en.wikipedia.org/wiki/All_models_are_wrong
> >
> >
> > On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
> > >   <
> https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16>
> > > Dear all,
> > >
> > > i would appreciate having your advice on the following please :
> > >
> > > in R, the wilcox.test() provides "a p-value < 2.2e-16", when we compare
> > > sets of 1000 genes expression (in the genomics field).
> > >
> > > however, the journal asks us to provide the exact p value ...
> > >
> > > would it be legitimate to write : "p-value = 0" ? thanks a lot,
> > >
> > > -- bogdan
> > >
> > >       [[alternative HTML version deleted]]
> > >
> > > ______________________________________________
> > > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > > and provide commented, minimal, self-contained, reproducible code.
> >
> > ______________________________________________
> > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Vivek Das
Hi Bogdan,

You can also get the information from the link of the Wilcox.test function
page.

“By default (if exact is not specified), an exact p-value is computed if
the samples contain less than 50 finite values and there are no ties.
Otherwise, a normal approximation is used.”

For more:

https://stat.ethz.ch/R-manual/R-devel/library/stats/html/wilcox.test.html

Hope this helps!

Best,

VD


On Thu, Mar 18, 2021 at 10:36 PM Bogdan Tanasa <[hidden email]> wrote:

> Dear Peter, thanks a lot. yes, we can see a very precise p-value, and that
> was the request from the journal.
>
> if I may ask another question please : what is the meaning of "exact=TRUE"
> or "exact=FALSE" in wilcox.test ?
>
> i can see that the "numerically precise" p-values are different. thanks a
> lot !
>
> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> tst$p.value
> [1] 8.535524e-25
>
> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=FALSE)
> tst$p.value
> [1] 3.448211e-25
>
> On Thu, Mar 18, 2021 at 10:15 PM Peter Langfelder <
> [hidden email]> wrote:
>
> > I thinnk the answer is much simpler. The print method for hypothesis
> > tests (class htest) truncates the p-values. In the above example,
> > instead of using
> >
> > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> >
> > and copying the output, just print the p-value:
> >
> > tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> > tst$p.value
> >
> > [1] 2.988368e-32
> >
> >
> > I think this value is what the journal asks for.
> >
> > HTH,
> >
> > Peter
> >
> > On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves
> > <[hidden email]> wrote:
> > >
> > >        I would push back on that from two perspectives:
> > >
> > >
> > >              1.  I would study exactly what the journal said very
> > > carefully.  If they mandated "wilcox.test", that function has an
> > > argument called "exact".  If that's what they are asking, then using
> > > that argument gives the exact p-value, e.g.:
> > >
> > >
> > >  > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> > >
> > >          Wilcoxon rank sum exact test
> > >
> > > data:  rnorm(100) and rnorm(100, 2)
> > > W = 691, p-value < 2.2e-16
> > >
> > >
> > >              2.  If that's NOT what they are asking, then I'm not
> > > convinced what they are asking makes sense:  There is is no such thing
> > > as an "exact p value" except to the extent that certain assumptions
> > > hold, and all models are wrong (but some are useful), as George Box
> > > famously said years ago.[1]  Truth only exists in mathematics, and
> > > that's because it's a fiction to start with ;-)
> > >
> > >
> > >        Hope this helps.
> > >        Spencer Graves
> > >
> > >
> > > [1]
> > > https://en.wikipedia.org/wiki/All_models_are_wrong
> > >
> > >
> > > On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
> > > >   <
> > https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16>
> > > > Dear all,
> > > >
> > > > i would appreciate having your advice on the following please :
> > > >
> > > > in R, the wilcox.test() provides "a p-value < 2.2e-16", when we
> compare
> > > > sets of 1000 genes expression (in the genomics field).
> > > >
> > > > however, the journal asks us to provide the exact p value ...
> > > >
> > > > would it be legitimate to write : "p-value = 0" ? thanks a lot,
> > > >
> > > > -- bogdan
> > > >
> > > >       [[alternative HTML version deleted]]
> > > >
> > > > ______________________________________________
> > > > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > > PLEASE do read the posting guide
> > http://www.R-project.org/posting-guide.html
> > > > and provide commented, minimal, self-contained, reproducible code.
> > >
> > > ______________________________________________
> > > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > PLEASE do read the posting guide
> > http://www.R-project.org/posting-guide.html
> > > and provide commented, minimal, self-contained, reproducible code.
> >
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
--
----------------------------------------------------------

Vivek Das, PhD

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Bogdan Tanasa
thanks a lot, Vivek ! in other words, assuming that we work with 1000 data
points,

shall we use EXACT = TRUE, it uses the normal approximation,

while if EXACT=FALSE (for these large samples), it does not ?

On Thu, Mar 18, 2021 at 10:47 PM Vivek Das <[hidden email]> wrote:

> Hi Bogdan,
>
> You can also get the information from the link of the Wilcox.test function
> page.
>
> “By default (if exact is not specified), an exact p-value is computed if
> the samples contain less than 50 finite values and there are no ties.
> Otherwise, a normal approximation is used.”
>
> For more:
>
> https://stat.ethz.ch/R-manual/R-devel/library/stats/html/wilcox.test.html
>
> Hope this helps!
>
> Best,
>
> VD
>
>
> On Thu, Mar 18, 2021 at 10:36 PM Bogdan Tanasa <[hidden email]> wrote:
>
>> Dear Peter, thanks a lot. yes, we can see a very precise p-value, and that
>> was the request from the journal.
>>
>> if I may ask another question please : what is the meaning of "exact=TRUE"
>> or "exact=FALSE" in wilcox.test ?
>>
>> i can see that the "numerically precise" p-values are different. thanks a
>> lot !
>>
>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>> tst$p.value
>> [1] 8.535524e-25
>>
>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=FALSE)
>> tst$p.value
>> [1] 3.448211e-25
>>
>> On Thu, Mar 18, 2021 at 10:15 PM Peter Langfelder <
>> [hidden email]> wrote:
>>
>> > I thinnk the answer is much simpler. The print method for hypothesis
>> > tests (class htest) truncates the p-values. In the above example,
>> > instead of using
>> >
>> > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>> >
>> > and copying the output, just print the p-value:
>> >
>> > tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>> > tst$p.value
>> >
>> > [1] 2.988368e-32
>> >
>> >
>> > I think this value is what the journal asks for.
>> >
>> > HTH,
>> >
>> > Peter
>> >
>> > On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves
>> > <[hidden email]> wrote:
>> > >
>> > >        I would push back on that from two perspectives:
>> > >
>> > >
>> > >              1.  I would study exactly what the journal said very
>> > > carefully.  If they mandated "wilcox.test", that function has an
>> > > argument called "exact".  If that's what they are asking, then using
>> > > that argument gives the exact p-value, e.g.:
>> > >
>> > >
>> > >  > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>> > >
>> > >          Wilcoxon rank sum exact test
>> > >
>> > > data:  rnorm(100) and rnorm(100, 2)
>> > > W = 691, p-value < 2.2e-16
>> > >
>> > >
>> > >              2.  If that's NOT what they are asking, then I'm not
>> > > convinced what they are asking makes sense:  There is is no such thing
>> > > as an "exact p value" except to the extent that certain assumptions
>> > > hold, and all models are wrong (but some are useful), as George Box
>> > > famously said years ago.[1]  Truth only exists in mathematics, and
>> > > that's because it's a fiction to start with ;-)
>> > >
>> > >
>> > >        Hope this helps.
>> > >        Spencer Graves
>> > >
>> > >
>> > > [1]
>> > > https://en.wikipedia.org/wiki/All_models_are_wrong
>> > >
>> > >
>> > > On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
>> > > >   <
>> > https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16
>> >
>> > > > Dear all,
>> > > >
>> > > > i would appreciate having your advice on the following please :
>> > > >
>> > > > in R, the wilcox.test() provides "a p-value < 2.2e-16", when we
>> compare
>> > > > sets of 1000 genes expression (in the genomics field).
>> > > >
>> > > > however, the journal asks us to provide the exact p value ...
>> > > >
>> > > > would it be legitimate to write : "p-value = 0" ? thanks a lot,
>> > > >
>> > > > -- bogdan
>> > > >
>> > > >       [[alternative HTML version deleted]]
>> > > >
>> > > > ______________________________________________
>> > > > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> > > > https://stat.ethz.ch/mailman/listinfo/r-help
>> > > > PLEASE do read the posting guide
>> > http://www.R-project.org/posting-guide.html
>> > > > and provide commented, minimal, self-contained, reproducible code.
>> > >
>> > > ______________________________________________
>> > > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> > > https://stat.ethz.ch/mailman/listinfo/r-help
>> > > PLEASE do read the posting guide
>> > http://www.R-project.org/posting-guide.html
>> > > and provide commented, minimal, self-contained, reproducible code.
>> >
>>
>>         [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
> --
> ----------------------------------------------------------
>
> Vivek Das, PhD
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

David Winsemius
In reply to this post by Bogdan Tanasa


Sent from my iPhone

> On Mar 18, 2021, at 10:26 PM, Bogdan Tanasa <[hidden email]> wrote:
>
> Dear Spencer, thank you very much for your prompt email and help. When
> using :
>
>> wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> W = 698, p-value < 2.2e-16
>
>> wilcox.test(rnorm(100), rnorm(100, 2), exact=FALSE)
> W = 1443, p-value < 2.2e-16
>
> and in both cases p-value < 2.2e-16. By "exact" p-value, i have meant the
> "precise" p-value ;
>
> If I may ask please, could we write p-value = 0 ?
>
> i have noted a similar conversation on stackexchange, although the answer
> is not very clear (to me).

The reason it wasn’t and couldn’t be “clear” was that the underlying scientific question and the statistical methods were not precisely described.

The same lack of background information still persists in this discussion.


David

>
> https://stats.stackexchange.com/questions/78839/how-should-tiny-p-values-be-reported-and-why-does-r-put-a-minimum-on-2-22e-1
>
> thanks again,
>
> bogdan
>
>> On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves <
>> [hidden email]> wrote:
>>      I would push back on that from two perspectives:
>>            1.  I would study exactly what the journal said very
>> carefully.  If they mandated "wilcox.test", that function has an
>> argument called "exact".  If that's what they are asking, then using
>> that argument gives the exact p-value, e.g.:
>>> wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>        Wilcoxon rank sum exact test
>> data:  rnorm(100) and rnorm(100, 2)
>> W = 691, p-value < 2.2e-16
>>            2.  If that's NOT what they are asking, then I'm not
>> convinced what they are asking makes sense:  There is is no such thing
>> as an "exact p value" except to the extent that certain assumptions
>> hold, and all models are wrong (but some are useful), as George Box
>> famously said years ago.[1]  Truth only exists in mathematics, and
>> that's because it's a fiction to start with ;-)
>>      Hope this helps.
>>      Spencer Graves
>> [1]
>> https://en.wikipedia.org/wiki/All_models_are_wrong
>>>> On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
>>>> <
>> https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16>
>>> Dear all,
>>> i would appreciate having your advice on the following please :
>>> in R, the wilcox.test() provides "a p-value < 2.2e-16", when we compare
>>> sets of 1000 genes expression (in the genomics field).
>>> however, the journal asks us to provide the exact p value ...
>>> would it be legitimate to write : "p-value = 0" ? thanks a lot,
>>> -- bogdan
>>>     [[alternative HTML version deleted]]
>>> ______________________________________________
>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>
>   [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Spencer Graves-4
In reply to this post by Bogdan Tanasa


On 2021-3-19 12:54 AM, Bogdan Tanasa wrote:
> thanks a lot, Vivek ! in other words, assuming that we work with 1000 data
> points,
>
> shall we use EXACT = TRUE, it uses the normal approximation,
>
> while if EXACT=FALSE (for these large samples), it does not ?


       As David Winsemius noted, the documentation is not clear.
Consider the following:

> set.seed(1)  > x <- rnorm(100) > y <- rnorm(100, 2) > > wilcox.test(x, y)$p.value
[1] 1.172189e-25 > wilcox.test(x, y)$p.value [1] 1.172189e-25 > >
wilcox.test(x, y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x,
y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
exact=TRUE)$p.value [1] 4.123875e-32 > wilcox.test(x, y,
exact=TRUE)$p.value [1] 4.123875e-32 > > wilcox.test(x, y,
EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
exact=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
exact=FALSE)$p.value [1] 1.172189e-25 > We get two values here:
1.172189e-25 and 4.123875e-32. The first one, I think, is the normal
approximation, which is the same as exact=FALSE. I think that with
exact=FALSE, you get a permutation distribution, though I'm not sure.
You might try looking at "wilcox_test in package coin for exact,
asymptotic and Monte Carlo conditional p-values, including in the
presence of ties" to see if it is clearer. NOTE: R is case sensitive, so
"EXACT" is a different variable from "exact". It is interpreted as an
optional argument, which is not recognized and therefore ignored in this
context.
          Hope this helps.
          Spencer


> On Thu, Mar 18, 2021 at 10:47 PM Vivek Das <[hidden email]> wrote:
>
>> Hi Bogdan,
>>
>> You can also get the information from the link of the Wilcox.test function
>> page.
>>
>> “By default (if exact is not specified), an exact p-value is computed if
>> the samples contain less than 50 finite values and there are no ties.
>> Otherwise, a normal approximation is used.”
>>
>> For more:
>>
>> https://stat.ethz.ch/R-manual/R-devel/library/stats/html/wilcox.test.html
>>
>> Hope this helps!
>>
>> Best,
>>
>> VD
>>
>>
>> On Thu, Mar 18, 2021 at 10:36 PM Bogdan Tanasa <[hidden email]> wrote:
>>
>>> Dear Peter, thanks a lot. yes, we can see a very precise p-value, and that
>>> was the request from the journal.
>>>
>>> if I may ask another question please : what is the meaning of "exact=TRUE"
>>> or "exact=FALSE" in wilcox.test ?
>>>
>>> i can see that the "numerically precise" p-values are different. thanks a
>>> lot !
>>>
>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>> tst$p.value
>>> [1] 8.535524e-25
>>>
>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=FALSE)
>>> tst$p.value
>>> [1] 3.448211e-25
>>>
>>> On Thu, Mar 18, 2021 at 10:15 PM Peter Langfelder <
>>> [hidden email]> wrote:
>>>
>>>> I thinnk the answer is much simpler. The print method for hypothesis
>>>> tests (class htest) truncates the p-values. In the above example,
>>>> instead of using
>>>>
>>>> wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>>>
>>>> and copying the output, just print the p-value:
>>>>
>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>>> tst$p.value
>>>>
>>>> [1] 2.988368e-32
>>>>
>>>>
>>>> I think this value is what the journal asks for.
>>>>
>>>> HTH,
>>>>
>>>> Peter
>>>>
>>>> On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves
>>>> <[hidden email]> wrote:
>>>>>         I would push back on that from two perspectives:
>>>>>
>>>>>
>>>>>               1.  I would study exactly what the journal said very
>>>>> carefully.  If they mandated "wilcox.test", that function has an
>>>>> argument called "exact".  If that's what they are asking, then using
>>>>> that argument gives the exact p-value, e.g.:
>>>>>
>>>>>
>>>>>   > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>>>>
>>>>>           Wilcoxon rank sum exact test
>>>>>
>>>>> data:  rnorm(100) and rnorm(100, 2)
>>>>> W = 691, p-value < 2.2e-16
>>>>>
>>>>>
>>>>>               2.  If that's NOT what they are asking, then I'm not
>>>>> convinced what they are asking makes sense:  There is is no such thing
>>>>> as an "exact p value" except to the extent that certain assumptions
>>>>> hold, and all models are wrong (but some are useful), as George Box
>>>>> famously said years ago.[1]  Truth only exists in mathematics, and
>>>>> that's because it's a fiction to start with ;-)
>>>>>
>>>>>
>>>>>         Hope this helps.
>>>>>         Spencer Graves
>>>>>
>>>>>
>>>>> [1]
>>>>> https://en.wikipedia.org/wiki/All_models_are_wrong
>>>>>
>>>>>
>>>>> On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
>>>>>>    <
>>>> https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16
>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> i would appreciate having your advice on the following please :
>>>>>>
>>>>>> in R, the wilcox.test() provides "a p-value < 2.2e-16", when we
>>> compare
>>>>>> sets of 1000 genes expression (in the genomics field).
>>>>>>
>>>>>> however, the journal asks us to provide the exact p value ...
>>>>>>
>>>>>> would it be legitimate to write : "p-value = 0" ? thanks a lot,
>>>>>>
>>>>>> -- bogdan
>>>>>>
>>>>>>        [[alternative HTML version deleted]]
>>>>>>
>>>>>> ______________________________________________
>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>>>> PLEASE do read the posting guide
>>>> http://www.R-project.org/posting-guide.html
>>>>>> and provide commented, minimal, self-contained, reproducible code.
>>>>> ______________________________________________
>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>>> PLEASE do read the posting guide
>>>> http://www.R-project.org/posting-guide.html
>>>>> and provide commented, minimal, self-contained, reproducible code.
>>>          [[alternative HTML version deleted]]
>>>
>>> ______________________________________________
>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>> --
>> ----------------------------------------------------------
>>
>> Vivek Das, PhD
>>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.


        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Jiefei Wang
Hey,

I just want to point out that the word "exact" has two meanings. It can
mean the numerically accurate p-value as Bogdan asked in his first email,
or it could mean the p-value calculated from the exact distribution of the
statistic(In this case, U stat). These two are actually not related, even
though they all called "exact".

Best,
Jiefei

On Fri, Mar 19, 2021 at 9:31 PM Spencer Graves <
[hidden email]> wrote:

>
>
> On 2021-3-19 12:54 AM, Bogdan Tanasa wrote:
> > thanks a lot, Vivek ! in other words, assuming that we work with 1000
> data
> > points,
> >
> > shall we use EXACT = TRUE, it uses the normal approximation,
> >
> > while if EXACT=FALSE (for these large samples), it does not ?
>
>
>        As David Winsemius noted, the documentation is not clear.
> Consider the following:
>
> > set.seed(1)  > x <- rnorm(100) > y <- rnorm(100, 2) > > wilcox.test(x,
> y)$p.value
> [1] 1.172189e-25 > wilcox.test(x, y)$p.value [1] 1.172189e-25 > >
> wilcox.test(x, y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x,
> y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
> exact=TRUE)$p.value [1] 4.123875e-32 > wilcox.test(x, y,
> exact=TRUE)$p.value [1] 4.123875e-32 > > wilcox.test(x, y,
> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
> exact=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
> exact=FALSE)$p.value [1] 1.172189e-25 > We get two values here:
> 1.172189e-25 and 4.123875e-32. The first one, I think, is the normal
> approximation, which is the same as exact=FALSE. I think that with
> exact=FALSE, you get a permutation distribution, though I'm not sure.
> You might try looking at "wilcox_test in package coin for exact,
> asymptotic and Monte Carlo conditional p-values, including in the
> presence of ties" to see if it is clearer. NOTE: R is case sensitive, so
> "EXACT" is a different variable from "exact". It is interpreted as an
> optional argument, which is not recognized and therefore ignored in this
> context.
>           Hope this helps.
>           Spencer
>
>
> > On Thu, Mar 18, 2021 at 10:47 PM Vivek Das <[hidden email]> wrote:
> >
> >> Hi Bogdan,
> >>
> >> You can also get the information from the link of the Wilcox.test
> function
> >> page.
> >>
> >> “By default (if exact is not specified), an exact p-value is computed if
> >> the samples contain less than 50 finite values and there are no ties.
> >> Otherwise, a normal approximation is used.”
> >>
> >> For more:
> >>
> >>
> https://stat.ethz.ch/R-manual/R-devel/library/stats/html/wilcox.test.html
> >>
> >> Hope this helps!
> >>
> >> Best,
> >>
> >> VD
> >>
> >>
> >> On Thu, Mar 18, 2021 at 10:36 PM Bogdan Tanasa <[hidden email]>
> wrote:
> >>
> >>> Dear Peter, thanks a lot. yes, we can see a very precise p-value, and
> that
> >>> was the request from the journal.
> >>>
> >>> if I may ask another question please : what is the meaning of
> "exact=TRUE"
> >>> or "exact=FALSE" in wilcox.test ?
> >>>
> >>> i can see that the "numerically precise" p-values are different.
> thanks a
> >>> lot !
> >>>
> >>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> >>> tst$p.value
> >>> [1] 8.535524e-25
> >>>
> >>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=FALSE)
> >>> tst$p.value
> >>> [1] 3.448211e-25
> >>>
> >>> On Thu, Mar 18, 2021 at 10:15 PM Peter Langfelder <
> >>> [hidden email]> wrote:
> >>>
> >>>> I thinnk the answer is much simpler. The print method for hypothesis
> >>>> tests (class htest) truncates the p-values. In the above example,
> >>>> instead of using
> >>>>
> >>>> wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> >>>>
> >>>> and copying the output, just print the p-value:
> >>>>
> >>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> >>>> tst$p.value
> >>>>
> >>>> [1] 2.988368e-32
> >>>>
> >>>>
> >>>> I think this value is what the journal asks for.
> >>>>
> >>>> HTH,
> >>>>
> >>>> Peter
> >>>>
> >>>> On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves
> >>>> <[hidden email]> wrote:
> >>>>>         I would push back on that from two perspectives:
> >>>>>
> >>>>>
> >>>>>               1.  I would study exactly what the journal said very
> >>>>> carefully.  If they mandated "wilcox.test", that function has an
> >>>>> argument called "exact".  If that's what they are asking, then using
> >>>>> that argument gives the exact p-value, e.g.:
> >>>>>
> >>>>>
> >>>>>   > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> >>>>>
> >>>>>           Wilcoxon rank sum exact test
> >>>>>
> >>>>> data:  rnorm(100) and rnorm(100, 2)
> >>>>> W = 691, p-value < 2.2e-16
> >>>>>
> >>>>>
> >>>>>               2.  If that's NOT what they are asking, then I'm not
> >>>>> convinced what they are asking makes sense:  There is is no such
> thing
> >>>>> as an "exact p value" except to the extent that certain assumptions
> >>>>> hold, and all models are wrong (but some are useful), as George Box
> >>>>> famously said years ago.[1]  Truth only exists in mathematics, and
> >>>>> that's because it's a fiction to start with ;-)
> >>>>>
> >>>>>
> >>>>>         Hope this helps.
> >>>>>         Spencer Graves
> >>>>>
> >>>>>
> >>>>> [1]
> >>>>> https://en.wikipedia.org/wiki/All_models_are_wrong
> >>>>>
> >>>>>
> >>>>> On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
> >>>>>>    <
> >>>>
> https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16
> >>>>
> >>>>>> Dear all,
> >>>>>>
> >>>>>> i would appreciate having your advice on the following please :
> >>>>>>
> >>>>>> in R, the wilcox.test() provides "a p-value < 2.2e-16", when we
> >>> compare
> >>>>>> sets of 1000 genes expression (in the genomics field).
> >>>>>>
> >>>>>> however, the journal asks us to provide the exact p value ...
> >>>>>>
> >>>>>> would it be legitimate to write : "p-value = 0" ? thanks a lot,
> >>>>>>
> >>>>>> -- bogdan
> >>>>>>
> >>>>>>        [[alternative HTML version deleted]]
> >>>>>>
> >>>>>> ______________________________________________
> >>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> >>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>>>>> PLEASE do read the posting guide
> >>>> http://www.R-project.org/posting-guide.html
> >>>>>> and provide commented, minimal, self-contained, reproducible code.
> >>>>> ______________________________________________
> >>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> >>>>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>>>> PLEASE do read the posting guide
> >>>> http://www.R-project.org/posting-guide.html
> >>>>> and provide commented, minimal, self-contained, reproducible code.
> >>>          [[alternative HTML version deleted]]
> >>>
> >>> ______________________________________________
> >>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> >>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>> PLEASE do read the posting guide
> >>> http://www.R-project.org/posting-guide.html
> >>> and provide commented, minimal, self-contained, reproducible code.
> >>>
> >> --
> >> ----------------------------------------------------------
> >>
> >> Vivek Das, PhD
> >>
> >       [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Jiefei Wang
After digging into the R source, it turns out that the argument `exact` has
nothing to do with the numeric precision. It only affects the statistic
model used to compute the p-value. When `exact=TRUE` the true distribution
of the statistic will be used. Otherwise, a normal approximation will be
used.

I think the documentation needs to be improved here, you can compute the
exact p-value *only* when you do not have any ties in your data. If you
have ties in your data you will get the p-value from the normal
approximation no matter what value you put in `exact`. This behavior should
be documented or a warning should be given when `exact=TRUE` and ties
present.

FYI, if the exact p-value is required, `pwilcox` function will be used to
compute the p-value. There are no details on how it computes the pvalue but
its C code seems to compute the probability table, so I assume it computes
the exact p-value from the true distribution of the statistic, not a
permutation or MC p-value.

Best,
Jiefei



On Fri, Mar 19, 2021 at 10:01 PM Jiefei Wang <[hidden email]> wrote:

> Hey,
>
> I just want to point out that the word "exact" has two meanings. It can
> mean the numerically accurate p-value as Bogdan asked in his first email,
> or it could mean the p-value calculated from the exact distribution of the
> statistic(In this case, U stat). These two are actually not related, even
> though they all called "exact".
>
> Best,
> Jiefei
>
> On Fri, Mar 19, 2021 at 9:31 PM Spencer Graves <
> [hidden email]> wrote:
>
>>
>>
>> On 2021-3-19 12:54 AM, Bogdan Tanasa wrote:
>> > thanks a lot, Vivek ! in other words, assuming that we work with 1000
>> data
>> > points,
>> >
>> > shall we use EXACT = TRUE, it uses the normal approximation,
>> >
>> > while if EXACT=FALSE (for these large samples), it does not ?
>>
>>
>>        As David Winsemius noted, the documentation is not clear.
>> Consider the following:
>>
>> > set.seed(1)  > x <- rnorm(100) > y <- rnorm(100, 2) > > wilcox.test(x,
>> y)$p.value
>> [1] 1.172189e-25 > wilcox.test(x, y)$p.value [1] 1.172189e-25 > >
>> wilcox.test(x, y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x,
>> y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>> exact=TRUE)$p.value [1] 4.123875e-32 > wilcox.test(x, y,
>> exact=TRUE)$p.value [1] 4.123875e-32 > > wilcox.test(x, y,
>> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>> exact=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>> exact=FALSE)$p.value [1] 1.172189e-25 > We get two values here:
>> 1.172189e-25 and 4.123875e-32. The first one, I think, is the normal
>> approximation, which is the same as exact=FALSE. I think that with
>> exact=FALSE, you get a permutation distribution, though I'm not sure.
>> You might try looking at "wilcox_test in package coin for exact,
>> asymptotic and Monte Carlo conditional p-values, including in the
>> presence of ties" to see if it is clearer. NOTE: R is case sensitive, so
>> "EXACT" is a different variable from "exact". It is interpreted as an
>> optional argument, which is not recognized and therefore ignored in this
>> context.
>>           Hope this helps.
>>           Spencer
>>
>>
>> > On Thu, Mar 18, 2021 at 10:47 PM Vivek Das <[hidden email]> wrote:
>> >
>> >> Hi Bogdan,
>> >>
>> >> You can also get the information from the link of the Wilcox.test
>> function
>> >> page.
>> >>
>> >> “By default (if exact is not specified), an exact p-value is computed
>> if
>> >> the samples contain less than 50 finite values and there are no ties.
>> >> Otherwise, a normal approximation is used.”
>> >>
>> >> For more:
>> >>
>> >>
>> https://stat.ethz.ch/R-manual/R-devel/library/stats/html/wilcox.test.html
>> >>
>> >> Hope this helps!
>> >>
>> >> Best,
>> >>
>> >> VD
>> >>
>> >>
>> >> On Thu, Mar 18, 2021 at 10:36 PM Bogdan Tanasa <[hidden email]>
>> wrote:
>> >>
>> >>> Dear Peter, thanks a lot. yes, we can see a very precise p-value, and
>> that
>> >>> was the request from the journal.
>> >>>
>> >>> if I may ask another question please : what is the meaning of
>> "exact=TRUE"
>> >>> or "exact=FALSE" in wilcox.test ?
>> >>>
>> >>> i can see that the "numerically precise" p-values are different.
>> thanks a
>> >>> lot !
>> >>>
>> >>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>> >>> tst$p.value
>> >>> [1] 8.535524e-25
>> >>>
>> >>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=FALSE)
>> >>> tst$p.value
>> >>> [1] 3.448211e-25
>> >>>
>> >>> On Thu, Mar 18, 2021 at 10:15 PM Peter Langfelder <
>> >>> [hidden email]> wrote:
>> >>>
>> >>>> I thinnk the answer is much simpler. The print method for hypothesis
>> >>>> tests (class htest) truncates the p-values. In the above example,
>> >>>> instead of using
>> >>>>
>> >>>> wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>> >>>>
>> >>>> and copying the output, just print the p-value:
>> >>>>
>> >>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>> >>>> tst$p.value
>> >>>>
>> >>>> [1] 2.988368e-32
>> >>>>
>> >>>>
>> >>>> I think this value is what the journal asks for.
>> >>>>
>> >>>> HTH,
>> >>>>
>> >>>> Peter
>> >>>>
>> >>>> On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves
>> >>>> <[hidden email]> wrote:
>> >>>>>         I would push back on that from two perspectives:
>> >>>>>
>> >>>>>
>> >>>>>               1.  I would study exactly what the journal said very
>> >>>>> carefully.  If they mandated "wilcox.test", that function has an
>> >>>>> argument called "exact".  If that's what they are asking, then using
>> >>>>> that argument gives the exact p-value, e.g.:
>> >>>>>
>> >>>>>
>> >>>>>   > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>> >>>>>
>> >>>>>           Wilcoxon rank sum exact test
>> >>>>>
>> >>>>> data:  rnorm(100) and rnorm(100, 2)
>> >>>>> W = 691, p-value < 2.2e-16
>> >>>>>
>> >>>>>
>> >>>>>               2.  If that's NOT what they are asking, then I'm not
>> >>>>> convinced what they are asking makes sense:  There is is no such
>> thing
>> >>>>> as an "exact p value" except to the extent that certain assumptions
>> >>>>> hold, and all models are wrong (but some are useful), as George Box
>> >>>>> famously said years ago.[1]  Truth only exists in mathematics, and
>> >>>>> that's because it's a fiction to start with ;-)
>> >>>>>
>> >>>>>
>> >>>>>         Hope this helps.
>> >>>>>         Spencer Graves
>> >>>>>
>> >>>>>
>> >>>>> [1]
>> >>>>> https://en.wikipedia.org/wiki/All_models_are_wrong
>> >>>>>
>> >>>>>
>> >>>>> On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
>> >>>>>>    <
>> >>>>
>> https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16
>> >>>>
>> >>>>>> Dear all,
>> >>>>>>
>> >>>>>> i would appreciate having your advice on the following please :
>> >>>>>>
>> >>>>>> in R, the wilcox.test() provides "a p-value < 2.2e-16", when we
>> >>> compare
>> >>>>>> sets of 1000 genes expression (in the genomics field).
>> >>>>>>
>> >>>>>> however, the journal asks us to provide the exact p value ...
>> >>>>>>
>> >>>>>> would it be legitimate to write : "p-value = 0" ? thanks a lot,
>> >>>>>>
>> >>>>>> -- bogdan
>> >>>>>>
>> >>>>>>        [[alternative HTML version deleted]]
>> >>>>>>
>> >>>>>> ______________________________________________
>> >>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> >>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>> >>>>>> PLEASE do read the posting guide
>> >>>> http://www.R-project.org/posting-guide.html
>> >>>>>> and provide commented, minimal, self-contained, reproducible code.
>> >>>>> ______________________________________________
>> >>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> >>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>> >>>>> PLEASE do read the posting guide
>> >>>> http://www.R-project.org/posting-guide.html
>> >>>>> and provide commented, minimal, self-contained, reproducible code.
>> >>>          [[alternative HTML version deleted]]
>> >>>
>> >>> ______________________________________________
>> >>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> >>> https://stat.ethz.ch/mailman/listinfo/r-help
>> >>> PLEASE do read the posting guide
>> >>> http://www.R-project.org/posting-guide.html
>> >>> and provide commented, minimal, self-contained, reproducible code.
>> >>>
>> >> --
>> >> ----------------------------------------------------------
>> >>
>> >> Vivek Das, PhD
>> >>
>> >       [[alternative HTML version deleted]]
>> >
>> > ______________________________________________
>> > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> > https://stat.ethz.ch/mailman/listinfo/r-help
>> > PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> > and provide commented, minimal, self-contained, reproducible code.
>>
>>
>>         [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Spencer Graves-4


On 2021-3-19 9:52 AM, Jiefei Wang wrote:

> After digging into the R source, it turns out that the argument `exact` has
> nothing to do with the numeric precision. It only affects the statistic
> model used to compute the p-value. When `exact=TRUE` the true distribution
> of the statistic will be used. Otherwise, a normal approximation will be
> used.
>
> I think the documentation needs to be improved here, you can compute the
> exact p-value *only* when you do not have any ties in your data. If you
> have ties in your data you will get the p-value from the normal
> approximation no matter what value you put in `exact`. This behavior should
> be documented or a warning should be given when `exact=TRUE` and ties
> present.
>
> FYI, if the exact p-value is required, `pwilcox` function will be used to
> compute the p-value. There are no details on how it computes the pvalue but
> its C code seems to compute the probability table, so I assume it computes
> the exact p-value from the true distribution of the statistic, not a
> permutation or MC p-value.


       My example shows that it does NOT use Monte Carlo, because
otherwise it uses some distribution.  I believe the term "exact" means
that it uses the permutation distribution, though I could be mistaken. 
If it's NOT a permutation distribution, I don't know what it is.


       Spencer

>
> Best,
> Jiefei
>
>
>
> On Fri, Mar 19, 2021 at 10:01 PM Jiefei Wang <[hidden email]> wrote:
>
>> Hey,
>>
>> I just want to point out that the word "exact" has two meanings. It can
>> mean the numerically accurate p-value as Bogdan asked in his first email,
>> or it could mean the p-value calculated from the exact distribution of the
>> statistic(In this case, U stat). These two are actually not related, even
>> though they all called "exact".
>>
>> Best,
>> Jiefei
>>
>> On Fri, Mar 19, 2021 at 9:31 PM Spencer Graves <
>> [hidden email]> wrote:
>>
>>>
>>> On 2021-3-19 12:54 AM, Bogdan Tanasa wrote:
>>>> thanks a lot, Vivek ! in other words, assuming that we work with 1000
>>> data
>>>> points,
>>>>
>>>> shall we use EXACT = TRUE, it uses the normal approximation,
>>>>
>>>> while if EXACT=FALSE (for these large samples), it does not ?
>>>
>>>         As David Winsemius noted, the documentation is not clear.
>>> Consider the following:
>>>
>>>> set.seed(1)  > x <- rnorm(100) > y <- rnorm(100, 2) > > wilcox.test(x,
>>> y)$p.value
>>> [1] 1.172189e-25 > wilcox.test(x, y)$p.value [1] 1.172189e-25 > >
>>> wilcox.test(x, y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x,
>>> y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>>> exact=TRUE)$p.value [1] 4.123875e-32 > wilcox.test(x, y,
>>> exact=TRUE)$p.value [1] 4.123875e-32 > > wilcox.test(x, y,
>>> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>>> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>>> exact=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>>> exact=FALSE)$p.value [1] 1.172189e-25 > We get two values here:
>>> 1.172189e-25 and 4.123875e-32. The first one, I think, is the normal
>>> approximation, which is the same as exact=FALSE. I think that with
>>> exact=FALSE, you get a permutation distribution, though I'm not sure.
>>> You might try looking at "wilcox_test in package coin for exact,
>>> asymptotic and Monte Carlo conditional p-values, including in the
>>> presence of ties" to see if it is clearer. NOTE: R is case sensitive, so
>>> "EXACT" is a different variable from "exact". It is interpreted as an
>>> optional argument, which is not recognized and therefore ignored in this
>>> context.
>>>            Hope this helps.
>>>            Spencer
>>>
>>>
>>>> On Thu, Mar 18, 2021 at 10:47 PM Vivek Das <[hidden email]> wrote:
>>>>
>>>>> Hi Bogdan,
>>>>>
>>>>> You can also get the information from the link of the Wilcox.test
>>> function
>>>>> page.
>>>>>
>>>>> “By default (if exact is not specified), an exact p-value is computed
>>> if
>>>>> the samples contain less than 50 finite values and there are no ties.
>>>>> Otherwise, a normal approximation is used.”
>>>>>
>>>>> For more:
>>>>>
>>>>>
>>> https://stat.ethz.ch/R-manual/R-devel/library/stats/html/wilcox.test.html
>>>>> Hope this helps!
>>>>>
>>>>> Best,
>>>>>
>>>>> VD
>>>>>
>>>>>
>>>>> On Thu, Mar 18, 2021 at 10:36 PM Bogdan Tanasa <[hidden email]>
>>> wrote:
>>>>>> Dear Peter, thanks a lot. yes, we can see a very precise p-value, and
>>> that
>>>>>> was the request from the journal.
>>>>>>
>>>>>> if I may ask another question please : what is the meaning of
>>> "exact=TRUE"
>>>>>> or "exact=FALSE" in wilcox.test ?
>>>>>>
>>>>>> i can see that the "numerically precise" p-values are different.
>>> thanks a
>>>>>> lot !
>>>>>>
>>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>>>>> tst$p.value
>>>>>> [1] 8.535524e-25
>>>>>>
>>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=FALSE)
>>>>>> tst$p.value
>>>>>> [1] 3.448211e-25
>>>>>>
>>>>>> On Thu, Mar 18, 2021 at 10:15 PM Peter Langfelder <
>>>>>> [hidden email]> wrote:
>>>>>>
>>>>>>> I thinnk the answer is much simpler. The print method for hypothesis
>>>>>>> tests (class htest) truncates the p-values. In the above example,
>>>>>>> instead of using
>>>>>>>
>>>>>>> wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>>>>>>
>>>>>>> and copying the output, just print the p-value:
>>>>>>>
>>>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>>>>>> tst$p.value
>>>>>>>
>>>>>>> [1] 2.988368e-32
>>>>>>>
>>>>>>>
>>>>>>> I think this value is what the journal asks for.
>>>>>>>
>>>>>>> HTH,
>>>>>>>
>>>>>>> Peter
>>>>>>>
>>>>>>> On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves
>>>>>>> <[hidden email]> wrote:
>>>>>>>>          I would push back on that from two perspectives:
>>>>>>>>
>>>>>>>>
>>>>>>>>                1.  I would study exactly what the journal said very
>>>>>>>> carefully.  If they mandated "wilcox.test", that function has an
>>>>>>>> argument called "exact".  If that's what they are asking, then using
>>>>>>>> that argument gives the exact p-value, e.g.:
>>>>>>>>
>>>>>>>>
>>>>>>>>    > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>>>>>>>
>>>>>>>>            Wilcoxon rank sum exact test
>>>>>>>>
>>>>>>>> data:  rnorm(100) and rnorm(100, 2)
>>>>>>>> W = 691, p-value < 2.2e-16
>>>>>>>>
>>>>>>>>
>>>>>>>>                2.  If that's NOT what they are asking, then I'm not
>>>>>>>> convinced what they are asking makes sense:  There is is no such
>>> thing
>>>>>>>> as an "exact p value" except to the extent that certain assumptions
>>>>>>>> hold, and all models are wrong (but some are useful), as George Box
>>>>>>>> famously said years ago.[1]  Truth only exists in mathematics, and
>>>>>>>> that's because it's a fiction to start with ;-)
>>>>>>>>
>>>>>>>>
>>>>>>>>          Hope this helps.
>>>>>>>>          Spencer Graves
>>>>>>>>
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://en.wikipedia.org/wiki/All_models_are_wrong
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
>>>>>>>>>     <
>>> https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16
>>>>>>>>> Dear all,
>>>>>>>>>
>>>>>>>>> i would appreciate having your advice on the following please :
>>>>>>>>>
>>>>>>>>> in R, the wilcox.test() provides "a p-value < 2.2e-16", when we
>>>>>> compare
>>>>>>>>> sets of 1000 genes expression (in the genomics field).
>>>>>>>>>
>>>>>>>>> however, the journal asks us to provide the exact p value ...
>>>>>>>>>
>>>>>>>>> would it be legitimate to write : "p-value = 0" ? thanks a lot,
>>>>>>>>>
>>>>>>>>> -- bogdan
>>>>>>>>>
>>>>>>>>>         [[alternative HTML version deleted]]
>>>>>>>>>
>>>>>>>>> ______________________________________________
>>>>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>>>>>>> PLEASE do read the posting guide
>>>>>>> http://www.R-project.org/posting-guide.html
>>>>>>>>> and provide commented, minimal, self-contained, reproducible code.
>>>>>>>> ______________________________________________
>>>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>>>>>> PLEASE do read the posting guide
>>>>>>> http://www.R-project.org/posting-guide.html
>>>>>>>> and provide commented, minimal, self-contained, reproducible code.
>>>>>>           [[alternative HTML version deleted]]
>>>>>>
>>>>>> ______________________________________________
>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>>>> PLEASE do read the posting guide
>>>>>> http://www.R-project.org/posting-guide.html
>>>>>> and provide commented, minimal, self-contained, reproducible code.
>>>>>>
>>>>> --
>>>>> ----------------------------------------------------------
>>>>>
>>>>> Vivek Das, PhD
>>>>>
>>>>        [[alternative HTML version deleted]]
>>>>
>>>> ______________________________________________
>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>>          [[alternative HTML version deleted]]
>>>
>>> ______________________________________________
>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Viechtbauer, Wolfgang (SP)
In reply to this post by Jiefei Wang
Dear Jiefei,

This behavior is documented. From help(wilcox.test):

"By default (if exact is not specified), an exact p-value is computed if the samples contain less than 50 finite values and there are no ties. Otherwise, a normal approximation is used."

Best,
Wolfgang

>-----Original Message-----
>From: R-help [mailto:[hidden email]] On Behalf Of Jiefei Wang
>Sent: Friday, 19 March, 2021 15:52
>To: Spencer Graves
>Cc: r-help; Bogdan Tanasa
>Subject: Re: [R] about a p-value < 2.2e-16
>
>After digging into the R source, it turns out that the argument `exact` has
>nothing to do with the numeric precision. It only affects the statistic
>model used to compute the p-value. When `exact=TRUE` the true distribution
>of the statistic will be used. Otherwise, a normal approximation will be
>used.
>
>I think the documentation needs to be improved here, you can compute the
>exact p-value *only* when you do not have any ties in your data. If you
>have ties in your data you will get the p-value from the normal
>approximation no matter what value you put in `exact`. This behavior should
>be documented or a warning should be given when `exact=TRUE` and ties
>present.
>
>FYI, if the exact p-value is required, `pwilcox` function will be used to
>compute the p-value. There are no details on how it computes the pvalue but
>its C code seems to compute the probability table, so I assume it computes
>the exact p-value from the true distribution of the statistic, not a
>permutation or MC p-value.
>
>Best,
>Jiefei
______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Jiefei Wang
Dear Wolfgang,

Thanks for the documentation, but the document only states the default
behavior, it does not mention what would happen if we tell it to compute
the exact p-value but the data has ties. I think this would be misleading
as people might think their result is exact by specifying `exact=TRUE` but
the truth is that their data contains ties and the result is from the
normal approximation.

Best,
Jiefei

On Fri, Mar 19, 2021 at 11:18 PM Viechtbauer, Wolfgang (SP) <
[hidden email]> wrote:

> Dear Jiefei,
>
> This behavior is documented. From help(wilcox.test):
>
> "By default (if exact is not specified), an exact p-value is computed if
> the samples contain less than 50 finite values and there are no ties.
> Otherwise, a normal approximation is used."
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: R-help [mailto:[hidden email]] On Behalf Of Jiefei
> Wang
> >Sent: Friday, 19 March, 2021 15:52
> >To: Spencer Graves
> >Cc: r-help; Bogdan Tanasa
> >Subject: Re: [R] about a p-value < 2.2e-16
> >
> >After digging into the R source, it turns out that the argument `exact`
> has
> >nothing to do with the numeric precision. It only affects the statistic
> >model used to compute the p-value. When `exact=TRUE` the true distribution
> >of the statistic will be used. Otherwise, a normal approximation will be
> >used.
> >
> >I think the documentation needs to be improved here, you can compute the
> >exact p-value *only* when you do not have any ties in your data. If you
> >have ties in your data you will get the p-value from the normal
> >approximation no matter what value you put in `exact`. This behavior
> should
> >be documented or a warning should be given when `exact=TRUE` and ties
> >present.
> >
> >FYI, if the exact p-value is required, `pwilcox` function will be used to
> >compute the p-value. There are no details on how it computes the pvalue
> but
> >its C code seems to compute the probability table, so I assume it computes
> >the exact p-value from the true distribution of the statistic, not a
> >permutation or MC p-value.
> >
> >Best,
> >Jiefei
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Bogdan Tanasa
In reply to this post by Jiefei Wang
Dear Jiefei, and all,

many thanks for your time and comments, suggestions, insights.

-- bogdan

On Fri, Mar 19, 2021 at 7:52 AM Jiefei Wang <[hidden email]> wrote:

> After digging into the R source, it turns out that the argument `exact`
> has nothing to do with the numeric precision. It only affects the statistic
> model used to compute the p-value. When `exact=TRUE` the true distribution
> of the statistic will be used. Otherwise, a normal approximation will be
> used.
>
> I think the documentation needs to be improved here, you can compute the
> exact p-value *only* when you do not have any ties in your data. If you
> have ties in your data you will get the p-value from the normal
> approximation no matter what value you put in `exact`. This behavior should
> be documented or a warning should be given when `exact=TRUE` and ties
> present.
>
> FYI, if the exact p-value is required, `pwilcox` function will be used to
> compute the p-value. There are no details on how it computes the pvalue but
> its C code seems to compute the probability table, so I assume it computes
> the exact p-value from the true distribution of the statistic, not a
> permutation or MC p-value.
>
> Best,
> Jiefei
>
>
>
> On Fri, Mar 19, 2021 at 10:01 PM Jiefei Wang <[hidden email]> wrote:
>
>> Hey,
>>
>> I just want to point out that the word "exact" has two meanings. It can
>> mean the numerically accurate p-value as Bogdan asked in his first email,
>> or it could mean the p-value calculated from the exact distribution of the
>> statistic(In this case, U stat). These two are actually not related, even
>> though they all called "exact".
>>
>> Best,
>> Jiefei
>>
>> On Fri, Mar 19, 2021 at 9:31 PM Spencer Graves <
>> [hidden email]> wrote:
>>
>>>
>>>
>>> On 2021-3-19 12:54 AM, Bogdan Tanasa wrote:
>>> > thanks a lot, Vivek ! in other words, assuming that we work with 1000
>>> data
>>> > points,
>>> >
>>> > shall we use EXACT = TRUE, it uses the normal approximation,
>>> >
>>> > while if EXACT=FALSE (for these large samples), it does not ?
>>>
>>>
>>>        As David Winsemius noted, the documentation is not clear.
>>> Consider the following:
>>>
>>> > set.seed(1)  > x <- rnorm(100) > y <- rnorm(100, 2) > > wilcox.test(x,
>>> y)$p.value
>>> [1] 1.172189e-25 > wilcox.test(x, y)$p.value [1] 1.172189e-25 > >
>>> wilcox.test(x, y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x,
>>> y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>>> exact=TRUE)$p.value [1] 4.123875e-32 > wilcox.test(x, y,
>>> exact=TRUE)$p.value [1] 4.123875e-32 > > wilcox.test(x, y,
>>> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>>> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>>> exact=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>>> exact=FALSE)$p.value [1] 1.172189e-25 > We get two values here:
>>> 1.172189e-25 and 4.123875e-32. The first one, I think, is the normal
>>> approximation, which is the same as exact=FALSE. I think that with
>>> exact=FALSE, you get a permutation distribution, though I'm not sure.
>>> You might try looking at "wilcox_test in package coin for exact,
>>> asymptotic and Monte Carlo conditional p-values, including in the
>>> presence of ties" to see if it is clearer. NOTE: R is case sensitive, so
>>> "EXACT" is a different variable from "exact". It is interpreted as an
>>> optional argument, which is not recognized and therefore ignored in this
>>> context.
>>>           Hope this helps.
>>>           Spencer
>>>
>>>
>>> > On Thu, Mar 18, 2021 at 10:47 PM Vivek Das <[hidden email]> wrote:
>>> >
>>> >> Hi Bogdan,
>>> >>
>>> >> You can also get the information from the link of the Wilcox.test
>>> function
>>> >> page.
>>> >>
>>> >> “By default (if exact is not specified), an exact p-value is computed
>>> if
>>> >> the samples contain less than 50 finite values and there are no ties.
>>> >> Otherwise, a normal approximation is used.”
>>> >>
>>> >> For more:
>>> >>
>>> >>
>>> https://stat.ethz.ch/R-manual/R-devel/library/stats/html/wilcox.test.html
>>> >>
>>> >> Hope this helps!
>>> >>
>>> >> Best,
>>> >>
>>> >> VD
>>> >>
>>> >>
>>> >> On Thu, Mar 18, 2021 at 10:36 PM Bogdan Tanasa <[hidden email]>
>>> wrote:
>>> >>
>>> >>> Dear Peter, thanks a lot. yes, we can see a very precise p-value,
>>> and that
>>> >>> was the request from the journal.
>>> >>>
>>> >>> if I may ask another question please : what is the meaning of
>>> "exact=TRUE"
>>> >>> or "exact=FALSE" in wilcox.test ?
>>> >>>
>>> >>> i can see that the "numerically precise" p-values are different.
>>> thanks a
>>> >>> lot !
>>> >>>
>>> >>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>> >>> tst$p.value
>>> >>> [1] 8.535524e-25
>>> >>>
>>> >>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=FALSE)
>>> >>> tst$p.value
>>> >>> [1] 3.448211e-25
>>> >>>
>>> >>> On Thu, Mar 18, 2021 at 10:15 PM Peter Langfelder <
>>> >>> [hidden email]> wrote:
>>> >>>
>>> >>>> I thinnk the answer is much simpler. The print method for hypothesis
>>> >>>> tests (class htest) truncates the p-values. In the above example,
>>> >>>> instead of using
>>> >>>>
>>> >>>> wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>> >>>>
>>> >>>> and copying the output, just print the p-value:
>>> >>>>
>>> >>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>> >>>> tst$p.value
>>> >>>>
>>> >>>> [1] 2.988368e-32
>>> >>>>
>>> >>>>
>>> >>>> I think this value is what the journal asks for.
>>> >>>>
>>> >>>> HTH,
>>> >>>>
>>> >>>> Peter
>>> >>>>
>>> >>>> On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves
>>> >>>> <[hidden email]> wrote:
>>> >>>>>         I would push back on that from two perspectives:
>>> >>>>>
>>> >>>>>
>>> >>>>>               1.  I would study exactly what the journal said very
>>> >>>>> carefully.  If they mandated "wilcox.test", that function has an
>>> >>>>> argument called "exact".  If that's what they are asking, then
>>> using
>>> >>>>> that argument gives the exact p-value, e.g.:
>>> >>>>>
>>> >>>>>
>>> >>>>>   > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>> >>>>>
>>> >>>>>           Wilcoxon rank sum exact test
>>> >>>>>
>>> >>>>> data:  rnorm(100) and rnorm(100, 2)
>>> >>>>> W = 691, p-value < 2.2e-16
>>> >>>>>
>>> >>>>>
>>> >>>>>               2.  If that's NOT what they are asking, then I'm not
>>> >>>>> convinced what they are asking makes sense:  There is is no such
>>> thing
>>> >>>>> as an "exact p value" except to the extent that certain assumptions
>>> >>>>> hold, and all models are wrong (but some are useful), as George Box
>>> >>>>> famously said years ago.[1]  Truth only exists in mathematics, and
>>> >>>>> that's because it's a fiction to start with ;-)
>>> >>>>>
>>> >>>>>
>>> >>>>>         Hope this helps.
>>> >>>>>         Spencer Graves
>>> >>>>>
>>> >>>>>
>>> >>>>> [1]
>>> >>>>> https://en.wikipedia.org/wiki/All_models_are_wrong
>>> >>>>>
>>> >>>>>
>>> >>>>> On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
>>> >>>>>>    <
>>> >>>>
>>> https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16
>>> >>>>
>>> >>>>>> Dear all,
>>> >>>>>>
>>> >>>>>> i would appreciate having your advice on the following please :
>>> >>>>>>
>>> >>>>>> in R, the wilcox.test() provides "a p-value < 2.2e-16", when we
>>> >>> compare
>>> >>>>>> sets of 1000 genes expression (in the genomics field).
>>> >>>>>>
>>> >>>>>> however, the journal asks us to provide the exact p value ...
>>> >>>>>>
>>> >>>>>> would it be legitimate to write : "p-value = 0" ? thanks a lot,
>>> >>>>>>
>>> >>>>>> -- bogdan
>>> >>>>>>
>>> >>>>>>        [[alternative HTML version deleted]]
>>> >>>>>>
>>> >>>>>> ______________________________________________
>>> >>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>> >>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> >>>>>> PLEASE do read the posting guide
>>> >>>> http://www.R-project.org/posting-guide.html
>>> >>>>>> and provide commented, minimal, self-contained, reproducible code.
>>> >>>>> ______________________________________________
>>> >>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>> >>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> >>>>> PLEASE do read the posting guide
>>> >>>> http://www.R-project.org/posting-guide.html
>>> >>>>> and provide commented, minimal, self-contained, reproducible code.
>>> >>>          [[alternative HTML version deleted]]
>>> >>>
>>> >>> ______________________________________________
>>> >>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>> >>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> >>> PLEASE do read the posting guide
>>> >>> http://www.R-project.org/posting-guide.html
>>> >>> and provide commented, minimal, self-contained, reproducible code.
>>> >>>
>>> >> --
>>> >> ----------------------------------------------------------
>>> >>
>>> >> Vivek Das, PhD
>>> >>
>>> >       [[alternative HTML version deleted]]
>>> >
>>> > ______________________________________________
>>> > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>> > https://stat.ethz.ch/mailman/listinfo/r-help
>>> > PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> > and provide commented, minimal, self-contained, reproducible code.
>>>
>>>
>>>         [[alternative HTML version deleted]]
>>>
>>> ______________________________________________
>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Jiefei Wang
In reply to this post by Spencer Graves-4
Hi Spencer,

Thanks for your test results, I do not know the answer as I haven't
used wilcox.test for many years. I do not know if it is possible to compute
the exact distribution of the Wilcoxon rank sum statistic, but I think it
is very likely, as the document of `Wilcoxon` says:

This distribution is obtained as follows. Let x and y be two random,
independent samples of size m and n. Then the Wilcoxon rank sum statistic
is the number of all pairs (x[i], y[j]) for which y[j] is not greater than
x[i]. This statistic takes values between 0 and m * n, and its mean and
variance are m * n / 2 and m * n * (m + n + 1) / 12, respectively.

As a nice feature of the non-parametric statistic, it is usually
distribution-free so you can pick any distribution you like to compute the
same statistic. I wonder if this is the case, but I might be wrong.

Cheers,
Jiefei


On Fri, Mar 19, 2021 at 10:57 PM Spencer Graves <
[hidden email]> wrote:

>
>
> On 2021-3-19 9:52 AM, Jiefei Wang wrote:
> > After digging into the R source, it turns out that the argument `exact`
> has
> > nothing to do with the numeric precision. It only affects the statistic
> > model used to compute the p-value. When `exact=TRUE` the true
> distribution
> > of the statistic will be used. Otherwise, a normal approximation will be
> > used.
> >
> > I think the documentation needs to be improved here, you can compute the
> > exact p-value *only* when you do not have any ties in your data. If you
> > have ties in your data you will get the p-value from the normal
> > approximation no matter what value you put in `exact`. This behavior
> should
> > be documented or a warning should be given when `exact=TRUE` and ties
> > present.
> >
> > FYI, if the exact p-value is required, `pwilcox` function will be used to
> > compute the p-value. There are no details on how it computes the pvalue
> but
> > its C code seems to compute the probability table, so I assume it
> computes
> > the exact p-value from the true distribution of the statistic, not a
> > permutation or MC p-value.
>
>
>        My example shows that it does NOT use Monte Carlo, because
> otherwise it uses some distribution.  I believe the term "exact" means
> that it uses the permutation distribution, though I could be mistaken.
> If it's NOT a permutation distribution, I don't know what it is.
>
>
>        Spencer
> >
> > Best,
> > Jiefei
> >
> >
> >
> > On Fri, Mar 19, 2021 at 10:01 PM Jiefei Wang <[hidden email]> wrote:
> >
> >> Hey,
> >>
> >> I just want to point out that the word "exact" has two meanings. It can
> >> mean the numerically accurate p-value as Bogdan asked in his first
> email,
> >> or it could mean the p-value calculated from the exact distribution of
> the
> >> statistic(In this case, U stat). These two are actually not related,
> even
> >> though they all called "exact".
> >>
> >> Best,
> >> Jiefei
> >>
> >> On Fri, Mar 19, 2021 at 9:31 PM Spencer Graves <
> >> [hidden email]> wrote:
> >>
> >>>
> >>> On 2021-3-19 12:54 AM, Bogdan Tanasa wrote:
> >>>> thanks a lot, Vivek ! in other words, assuming that we work with 1000
> >>> data
> >>>> points,
> >>>>
> >>>> shall we use EXACT = TRUE, it uses the normal approximation,
> >>>>
> >>>> while if EXACT=FALSE (for these large samples), it does not ?
> >>>
> >>>         As David Winsemius noted, the documentation is not clear.
> >>> Consider the following:
> >>>
> >>>> set.seed(1)  > x <- rnorm(100) > y <- rnorm(100, 2) > > wilcox.test(x,
> >>> y)$p.value
> >>> [1] 1.172189e-25 > wilcox.test(x, y)$p.value [1] 1.172189e-25 > >
> >>> wilcox.test(x, y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x,
> >>> y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
> >>> exact=TRUE)$p.value [1] 4.123875e-32 > wilcox.test(x, y,
> >>> exact=TRUE)$p.value [1] 4.123875e-32 > > wilcox.test(x, y,
> >>> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
> >>> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
> >>> exact=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
> >>> exact=FALSE)$p.value [1] 1.172189e-25 > We get two values here:
> >>> 1.172189e-25 and 4.123875e-32. The first one, I think, is the normal
> >>> approximation, which is the same as exact=FALSE. I think that with
> >>> exact=FALSE, you get a permutation distribution, though I'm not sure.
> >>> You might try looking at "wilcox_test in package coin for exact,
> >>> asymptotic and Monte Carlo conditional p-values, including in the
> >>> presence of ties" to see if it is clearer. NOTE: R is case sensitive,
> so
> >>> "EXACT" is a different variable from "exact". It is interpreted as an
> >>> optional argument, which is not recognized and therefore ignored in
> this
> >>> context.
> >>>            Hope this helps.
> >>>            Spencer
> >>>
> >>>
> >>>> On Thu, Mar 18, 2021 at 10:47 PM Vivek Das <[hidden email]>
> wrote:
> >>>>
> >>>>> Hi Bogdan,
> >>>>>
> >>>>> You can also get the information from the link of the Wilcox.test
> >>> function
> >>>>> page.
> >>>>>
> >>>>> “By default (if exact is not specified), an exact p-value is computed
> >>> if
> >>>>> the samples contain less than 50 finite values and there are no ties.
> >>>>> Otherwise, a normal approximation is used.”
> >>>>>
> >>>>> For more:
> >>>>>
> >>>>>
> >>>
> https://stat.ethz.ch/R-manual/R-devel/library/stats/html/wilcox.test.html
> >>>>> Hope this helps!
> >>>>>
> >>>>> Best,
> >>>>>
> >>>>> VD
> >>>>>
> >>>>>
> >>>>> On Thu, Mar 18, 2021 at 10:36 PM Bogdan Tanasa <[hidden email]>
> >>> wrote:
> >>>>>> Dear Peter, thanks a lot. yes, we can see a very precise p-value,
> and
> >>> that
> >>>>>> was the request from the journal.
> >>>>>>
> >>>>>> if I may ask another question please : what is the meaning of
> >>> "exact=TRUE"
> >>>>>> or "exact=FALSE" in wilcox.test ?
> >>>>>>
> >>>>>> i can see that the "numerically precise" p-values are different.
> >>> thanks a
> >>>>>> lot !
> >>>>>>
> >>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> >>>>>> tst$p.value
> >>>>>> [1] 8.535524e-25
> >>>>>>
> >>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=FALSE)
> >>>>>> tst$p.value
> >>>>>> [1] 3.448211e-25
> >>>>>>
> >>>>>> On Thu, Mar 18, 2021 at 10:15 PM Peter Langfelder <
> >>>>>> [hidden email]> wrote:
> >>>>>>
> >>>>>>> I thinnk the answer is much simpler. The print method for
> hypothesis
> >>>>>>> tests (class htest) truncates the p-values. In the above example,
> >>>>>>> instead of using
> >>>>>>>
> >>>>>>> wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> >>>>>>>
> >>>>>>> and copying the output, just print the p-value:
> >>>>>>>
> >>>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> >>>>>>> tst$p.value
> >>>>>>>
> >>>>>>> [1] 2.988368e-32
> >>>>>>>
> >>>>>>>
> >>>>>>> I think this value is what the journal asks for.
> >>>>>>>
> >>>>>>> HTH,
> >>>>>>>
> >>>>>>> Peter
> >>>>>>>
> >>>>>>> On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves
> >>>>>>> <[hidden email]> wrote:
> >>>>>>>>          I would push back on that from two perspectives:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>                1.  I would study exactly what the journal said
> very
> >>>>>>>> carefully.  If they mandated "wilcox.test", that function has an
> >>>>>>>> argument called "exact".  If that's what they are asking, then
> using
> >>>>>>>> that argument gives the exact p-value, e.g.:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>    > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> >>>>>>>>
> >>>>>>>>            Wilcoxon rank sum exact test
> >>>>>>>>
> >>>>>>>> data:  rnorm(100) and rnorm(100, 2)
> >>>>>>>> W = 691, p-value < 2.2e-16
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>                2.  If that's NOT what they are asking, then I'm
> not
> >>>>>>>> convinced what they are asking makes sense:  There is is no such
> >>> thing
> >>>>>>>> as an "exact p value" except to the extent that certain
> assumptions
> >>>>>>>> hold, and all models are wrong (but some are useful), as George
> Box
> >>>>>>>> famously said years ago.[1]  Truth only exists in mathematics, and
> >>>>>>>> that's because it's a fiction to start with ;-)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>          Hope this helps.
> >>>>>>>>          Spencer Graves
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>> https://en.wikipedia.org/wiki/All_models_are_wrong
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
> >>>>>>>>>     <
> >>>
> https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16
> >>>>>>>>> Dear all,
> >>>>>>>>>
> >>>>>>>>> i would appreciate having your advice on the following please :
> >>>>>>>>>
> >>>>>>>>> in R, the wilcox.test() provides "a p-value < 2.2e-16", when we
> >>>>>> compare
> >>>>>>>>> sets of 1000 genes expression (in the genomics field).
> >>>>>>>>>
> >>>>>>>>> however, the journal asks us to provide the exact p value ...
> >>>>>>>>>
> >>>>>>>>> would it be legitimate to write : "p-value = 0" ? thanks a lot,
> >>>>>>>>>
> >>>>>>>>> -- bogdan
> >>>>>>>>>
> >>>>>>>>>         [[alternative HTML version deleted]]
> >>>>>>>>>
> >>>>>>>>> ______________________________________________
> >>>>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more,
> see
> >>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>>>>>>>> PLEASE do read the posting guide
> >>>>>>> http://www.R-project.org/posting-guide.html
> >>>>>>>>> and provide commented, minimal, self-contained, reproducible
> code.
> >>>>>>>> ______________________________________________
> >>>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> >>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>>>>>>> PLEASE do read the posting guide
> >>>>>>> http://www.R-project.org/posting-guide.html
> >>>>>>>> and provide commented, minimal, self-contained, reproducible code.
> >>>>>>           [[alternative HTML version deleted]]
> >>>>>>
> >>>>>> ______________________________________________
> >>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> >>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>>>>> PLEASE do read the posting guide
> >>>>>> http://www.R-project.org/posting-guide.html
> >>>>>> and provide commented, minimal, self-contained, reproducible code.
> >>>>>>
> >>>>> --
> >>>>> ----------------------------------------------------------
> >>>>>
> >>>>> Vivek Das, PhD
> >>>>>
> >>>>        [[alternative HTML version deleted]]
> >>>>
> >>>> ______________________________________________
> >>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> >>>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>>> PLEASE do read the posting guide
> >>> http://www.R-project.org/posting-guide.html
> >>>> and provide commented, minimal, self-contained, reproducible code.
> >>>
> >>>          [[alternative HTML version deleted]]
> >>>
> >>> ______________________________________________
> >>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> >>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>> PLEASE do read the posting guide
> >>> http://www.R-project.org/posting-guide.html
> >>> and provide commented, minimal, self-contained, reproducible code.
> >>>
>
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Bert Gunter-2
I **believe** -- if my old memory still serves-- that the "exact"
specification uses a home grown version of the algorithm to calculate
exact,  or close approximations to the exact, permutation distribution
originally developed by Cyrus Mehta, founder of StatXact software.  Of
course, examining the C code source would determine this, but I don't care
to attempt this.

If this is (no longer?) correct, please point this out.

Best,

Bert Gunter

"The trouble with having an open mind is that people keep coming along and
sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )


On Fri, Mar 19, 2021 at 8:42 AM Jiefei Wang <[hidden email]> wrote:

> Hi Spencer,
>
> Thanks for your test results, I do not know the answer as I haven't
> used wilcox.test for many years. I do not know if it is possible to compute
> the exact distribution of the Wilcoxon rank sum statistic, but I think it
> is very likely, as the document of `Wilcoxon` says:
>
> This distribution is obtained as follows. Let x and y be two random,
> independent samples of size m and n. Then the Wilcoxon rank sum statistic
> is the number of all pairs (x[i], y[j]) for which y[j] is not greater than
> x[i]. This statistic takes values between 0 and m * n, and its mean and
> variance are m * n / 2 and m * n * (m + n + 1) / 12, respectively.
>
> As a nice feature of the non-parametric statistic, it is usually
> distribution-free so you can pick any distribution you like to compute the
> same statistic. I wonder if this is the case, but I might be wrong.
>
> Cheers,
> Jiefei
>
>
> On Fri, Mar 19, 2021 at 10:57 PM Spencer Graves <
> [hidden email]> wrote:
>
> >
> >
> > On 2021-3-19 9:52 AM, Jiefei Wang wrote:
> > > After digging into the R source, it turns out that the argument `exact`
> > has
> > > nothing to do with the numeric precision. It only affects the statistic
> > > model used to compute the p-value. When `exact=TRUE` the true
> > distribution
> > > of the statistic will be used. Otherwise, a normal approximation will
> be
> > > used.
> > >
> > > I think the documentation needs to be improved here, you can compute
> the
> > > exact p-value *only* when you do not have any ties in your data. If you
> > > have ties in your data you will get the p-value from the normal
> > > approximation no matter what value you put in `exact`. This behavior
> > should
> > > be documented or a warning should be given when `exact=TRUE` and ties
> > > present.
> > >
> > > FYI, if the exact p-value is required, `pwilcox` function will be used
> to
> > > compute the p-value. There are no details on how it computes the pvalue
> > but
> > > its C code seems to compute the probability table, so I assume it
> > computes
> > > the exact p-value from the true distribution of the statistic, not a
> > > permutation or MC p-value.
> >
> >
> >        My example shows that it does NOT use Monte Carlo, because
> > otherwise it uses some distribution.  I believe the term "exact" means
> > that it uses the permutation distribution, though I could be mistaken.
> > If it's NOT a permutation distribution, I don't know what it is.
> >
> >
> >        Spencer
> > >
> > > Best,
> > > Jiefei
> > >
> > >
> > >
> > > On Fri, Mar 19, 2021 at 10:01 PM Jiefei Wang <[hidden email]>
> wrote:
> > >
> > >> Hey,
> > >>
> > >> I just want to point out that the word "exact" has two meanings. It
> can
> > >> mean the numerically accurate p-value as Bogdan asked in his first
> > email,
> > >> or it could mean the p-value calculated from the exact distribution of
> > the
> > >> statistic(In this case, U stat). These two are actually not related,
> > even
> > >> though they all called "exact".
> > >>
> > >> Best,
> > >> Jiefei
> > >>
> > >> On Fri, Mar 19, 2021 at 9:31 PM Spencer Graves <
> > >> [hidden email]> wrote:
> > >>
> > >>>
> > >>> On 2021-3-19 12:54 AM, Bogdan Tanasa wrote:
> > >>>> thanks a lot, Vivek ! in other words, assuming that we work with
> 1000
> > >>> data
> > >>>> points,
> > >>>>
> > >>>> shall we use EXACT = TRUE, it uses the normal approximation,
> > >>>>
> > >>>> while if EXACT=FALSE (for these large samples), it does not ?
> > >>>
> > >>>         As David Winsemius noted, the documentation is not clear.
> > >>> Consider the following:
> > >>>
> > >>>> set.seed(1)  > x <- rnorm(100) > y <- rnorm(100, 2) > >
> wilcox.test(x,
> > >>> y)$p.value
> > >>> [1] 1.172189e-25 > wilcox.test(x, y)$p.value [1] 1.172189e-25 > >
> > >>> wilcox.test(x, y, EXACT=TRUE)$p.value [1] 1.172189e-25 >
> wilcox.test(x,
> > >>> y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
> > >>> exact=TRUE)$p.value [1] 4.123875e-32 > wilcox.test(x, y,
> > >>> exact=TRUE)$p.value [1] 4.123875e-32 > > wilcox.test(x, y,
> > >>> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
> > >>> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
> > >>> exact=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
> > >>> exact=FALSE)$p.value [1] 1.172189e-25 > We get two values here:
> > >>> 1.172189e-25 and 4.123875e-32. The first one, I think, is the normal
> > >>> approximation, which is the same as exact=FALSE. I think that with
> > >>> exact=FALSE, you get a permutation distribution, though I'm not sure.
> > >>> You might try looking at "wilcox_test in package coin for exact,
> > >>> asymptotic and Monte Carlo conditional p-values, including in the
> > >>> presence of ties" to see if it is clearer. NOTE: R is case sensitive,
> > so
> > >>> "EXACT" is a different variable from "exact". It is interpreted as an
> > >>> optional argument, which is not recognized and therefore ignored in
> > this
> > >>> context.
> > >>>            Hope this helps.
> > >>>            Spencer
> > >>>
> > >>>
> > >>>> On Thu, Mar 18, 2021 at 10:47 PM Vivek Das <[hidden email]>
> > wrote:
> > >>>>
> > >>>>> Hi Bogdan,
> > >>>>>
> > >>>>> You can also get the information from the link of the Wilcox.test
> > >>> function
> > >>>>> page.
> > >>>>>
> > >>>>> “By default (if exact is not specified), an exact p-value is
> computed
> > >>> if
> > >>>>> the samples contain less than 50 finite values and there are no
> ties.
> > >>>>> Otherwise, a normal approximation is used.”
> > >>>>>
> > >>>>> For more:
> > >>>>>
> > >>>>>
> > >>>
> >
> https://stat.ethz.ch/R-manual/R-devel/library/stats/html/wilcox.test.html
> > >>>>> Hope this helps!
> > >>>>>
> > >>>>> Best,
> > >>>>>
> > >>>>> VD
> > >>>>>
> > >>>>>
> > >>>>> On Thu, Mar 18, 2021 at 10:36 PM Bogdan Tanasa <[hidden email]>
> > >>> wrote:
> > >>>>>> Dear Peter, thanks a lot. yes, we can see a very precise p-value,
> > and
> > >>> that
> > >>>>>> was the request from the journal.
> > >>>>>>
> > >>>>>> if I may ask another question please : what is the meaning of
> > >>> "exact=TRUE"
> > >>>>>> or "exact=FALSE" in wilcox.test ?
> > >>>>>>
> > >>>>>> i can see that the "numerically precise" p-values are different.
> > >>> thanks a
> > >>>>>> lot !
> > >>>>>>
> > >>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> > >>>>>> tst$p.value
> > >>>>>> [1] 8.535524e-25
> > >>>>>>
> > >>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=FALSE)
> > >>>>>> tst$p.value
> > >>>>>> [1] 3.448211e-25
> > >>>>>>
> > >>>>>> On Thu, Mar 18, 2021 at 10:15 PM Peter Langfelder <
> > >>>>>> [hidden email]> wrote:
> > >>>>>>
> > >>>>>>> I thinnk the answer is much simpler. The print method for
> > hypothesis
> > >>>>>>> tests (class htest) truncates the p-values. In the above example,
> > >>>>>>> instead of using
> > >>>>>>>
> > >>>>>>> wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> > >>>>>>>
> > >>>>>>> and copying the output, just print the p-value:
> > >>>>>>>
> > >>>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> > >>>>>>> tst$p.value
> > >>>>>>>
> > >>>>>>> [1] 2.988368e-32
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> I think this value is what the journal asks for.
> > >>>>>>>
> > >>>>>>> HTH,
> > >>>>>>>
> > >>>>>>> Peter
> > >>>>>>>
> > >>>>>>> On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves
> > >>>>>>> <[hidden email]> wrote:
> > >>>>>>>>          I would push back on that from two perspectives:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>                1.  I would study exactly what the journal said
> > very
> > >>>>>>>> carefully.  If they mandated "wilcox.test", that function has an
> > >>>>>>>> argument called "exact".  If that's what they are asking, then
> > using
> > >>>>>>>> that argument gives the exact p-value, e.g.:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>    > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
> > >>>>>>>>
> > >>>>>>>>            Wilcoxon rank sum exact test
> > >>>>>>>>
> > >>>>>>>> data:  rnorm(100) and rnorm(100, 2)
> > >>>>>>>> W = 691, p-value < 2.2e-16
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>                2.  If that's NOT what they are asking, then I'm
> > not
> > >>>>>>>> convinced what they are asking makes sense:  There is is no such
> > >>> thing
> > >>>>>>>> as an "exact p value" except to the extent that certain
> > assumptions
> > >>>>>>>> hold, and all models are wrong (but some are useful), as George
> > Box
> > >>>>>>>> famously said years ago.[1]  Truth only exists in mathematics,
> and
> > >>>>>>>> that's because it's a fiction to start with ;-)
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>          Hope this helps.
> > >>>>>>>>          Spencer Graves
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> [1]
> > >>>>>>>> https://en.wikipedia.org/wiki/All_models_are_wrong
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
> > >>>>>>>>>     <
> > >>>
> > https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16
> > >>>>>>>>> Dear all,
> > >>>>>>>>>
> > >>>>>>>>> i would appreciate having your advice on the following please :
> > >>>>>>>>>
> > >>>>>>>>> in R, the wilcox.test() provides "a p-value < 2.2e-16", when we
> > >>>>>> compare
> > >>>>>>>>> sets of 1000 genes expression (in the genomics field).
> > >>>>>>>>>
> > >>>>>>>>> however, the journal asks us to provide the exact p value ...
> > >>>>>>>>>
> > >>>>>>>>> would it be legitimate to write : "p-value = 0" ? thanks a lot,
> > >>>>>>>>>
> > >>>>>>>>> -- bogdan
> > >>>>>>>>>
> > >>>>>>>>>         [[alternative HTML version deleted]]
> > >>>>>>>>>
> > >>>>>>>>> ______________________________________________
> > >>>>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more,
> > see
> > >>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
> > >>>>>>>>> PLEASE do read the posting guide
> > >>>>>>> http://www.R-project.org/posting-guide.html
> > >>>>>>>>> and provide commented, minimal, self-contained, reproducible
> > code.
> > >>>>>>>> ______________________________________________
> > >>>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more,
> see
> > >>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
> > >>>>>>>> PLEASE do read the posting guide
> > >>>>>>> http://www.R-project.org/posting-guide.html
> > >>>>>>>> and provide commented, minimal, self-contained, reproducible
> code.
> > >>>>>>           [[alternative HTML version deleted]]
> > >>>>>>
> > >>>>>> ______________________________________________
> > >>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > >>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
> > >>>>>> PLEASE do read the posting guide
> > >>>>>> http://www.R-project.org/posting-guide.html
> > >>>>>> and provide commented, minimal, self-contained, reproducible code.
> > >>>>>>
> > >>>>> --
> > >>>>> ----------------------------------------------------------
> > >>>>>
> > >>>>> Vivek Das, PhD
> > >>>>>
> > >>>>        [[alternative HTML version deleted]]
> > >>>>
> > >>>> ______________________________________________
> > >>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > >>>> https://stat.ethz.ch/mailman/listinfo/r-help
> > >>>> PLEASE do read the posting guide
> > >>> http://www.R-project.org/posting-guide.html
> > >>>> and provide commented, minimal, self-contained, reproducible code.
> > >>>
> > >>>          [[alternative HTML version deleted]]
> > >>>
> > >>> ______________________________________________
> > >>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > >>> https://stat.ethz.ch/mailman/listinfo/r-help
> > >>> PLEASE do read the posting guide
> > >>> http://www.R-project.org/posting-guide.html
> > >>> and provide commented, minimal, self-contained, reproducible code.
> > >>>
> >
> >
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Viechtbauer, Wolfgang (SP)
In reply to this post by Jiefei Wang
For me, it was always clear based on the documentation that if there are ties, then the normal approximation is used (irrespective of what 'exact' is set to). In fact, if there are ties, the output even tells you that this is happening:

wilcox.test(c(1,3,2,2,4), exact=TRUE)

[...]
Warning message:
In wilcox.test.default(c(1, 3, 2, 2, 4), exact = TRUE) :
  cannot compute exact p-value with ties

Best,
Wolfgang

>-----Original Message-----
>From: Jiefei Wang [mailto:[hidden email]]
>Sent: Friday, 19 March, 2021 16:32
>To: Viechtbauer, Wolfgang (SP)
>Cc: r-help
>Subject: Re: [R] about a p-value < 2.2e-16
>
>Dear Wolfgang,
>
>Thanks for the documentation, but the document only states the default behavior,
>it does not mention what would happen if we tell it to compute the exact p-value
>but the data has ties. I think this would be misleading as people might think
>their result is exact by specifying `exact=TRUE` but the truth is that their data
>contains ties and the result is from the normal approximation.
>
>Best,
>Jiefei
>
>On Fri, Mar 19, 2021 at 11:18 PM Viechtbauer, Wolfgang (SP)
><[hidden email]> wrote:
>Dear Jiefei,
>
>This behavior is documented. From help(wilcox.test):
>
>"By default (if exact is not specified), an exact p-value is computed if the
>samples contain less than 50 finite values and there are no ties. Otherwise, a
>normal approximation is used."
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: R-help [mailto:[hidden email]] On Behalf Of Jiefei Wang
>>Sent: Friday, 19 March, 2021 15:52
>>To: Spencer Graves
>>Cc: r-help; Bogdan Tanasa
>>Subject: Re: [R] about a p-value < 2.2e-16
>>
>>After digging into the R source, it turns out that the argument `exact` has
>>nothing to do with the numeric precision. It only affects the statistic
>>model used to compute the p-value. When `exact=TRUE` the true distribution
>>of the statistic will be used. Otherwise, a normal approximation will be
>>used.
>>
>>I think the documentation needs to be improved here, you can compute the
>>exact p-value *only* when you do not have any ties in your data. If you
>>have ties in your data you will get the p-value from the normal
>>approximation no matter what value you put in `exact`. This behavior should
>>be documented or a warning should be given when `exact=TRUE` and ties
>>present.
>>
>>FYI, if the exact p-value is required, `pwilcox` function will be used to
>>compute the p-value. There are no details on how it computes the pvalue but
>>its C code seems to compute the probability table, so I assume it computes
>>the exact p-value from the true distribution of the statistic, not a
>>permutation or MC p-value.
>>
>>Best,
>>Jiefei
______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Bogdan Tanasa
In reply to this post by Bert Gunter-2
Dear all, thank you all for comments and help.

as far as i can see, shall we have samples of 1000 records, only
"exact=FALSE" allows the code to run:

wilcox.test(rnorm(1000), rnorm(1000, 2), exact=FALSE)$p.value
[1] 7.304863e-231

shall i use "exact=TRUE", it runs out of memory on my 64GB RAM PC :

wilcox.test(rnorm(1000), rnorm(1000, 2), exact=TRUE)$p.value
(the job is terminated by OS)

shall you have any other suggestions, please let me know. thanks a lot !

On Fri, Mar 19, 2021 at 9:05 AM Bert Gunter <[hidden email]> wrote:

> I **believe** -- if my old memory still serves-- that the "exact"
> specification uses a home grown version of the algorithm to calculate
> exact,  or close approximations to the exact, permutation distribution
> originally developed by Cyrus Mehta, founder of StatXact software.  Of
> course, examining the C code source would determine this, but I don't care
> to attempt this.
>
> If this is (no longer?) correct, please point this out.
>
> Best,
>
> Bert Gunter
>
> "The trouble with having an open mind is that people keep coming along and
> sticking things into it."
> -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )
>
>
> On Fri, Mar 19, 2021 at 8:42 AM Jiefei Wang <[hidden email]> wrote:
>
>> Hi Spencer,
>>
>> Thanks for your test results, I do not know the answer as I haven't
>> used wilcox.test for many years. I do not know if it is possible to
>> compute
>> the exact distribution of the Wilcoxon rank sum statistic, but I think it
>> is very likely, as the document of `Wilcoxon` says:
>>
>> This distribution is obtained as follows. Let x and y be two random,
>> independent samples of size m and n. Then the Wilcoxon rank sum statistic
>> is the number of all pairs (x[i], y[j]) for which y[j] is not greater than
>> x[i]. This statistic takes values between 0 and m * n, and its mean and
>> variance are m * n / 2 and m * n * (m + n + 1) / 12, respectively.
>>
>> As a nice feature of the non-parametric statistic, it is usually
>> distribution-free so you can pick any distribution you like to compute the
>> same statistic. I wonder if this is the case, but I might be wrong.
>>
>> Cheers,
>> Jiefei
>>
>>
>> On Fri, Mar 19, 2021 at 10:57 PM Spencer Graves <
>> [hidden email]> wrote:
>>
>> >
>> >
>> > On 2021-3-19 9:52 AM, Jiefei Wang wrote:
>> > > After digging into the R source, it turns out that the argument
>> `exact`
>> > has
>> > > nothing to do with the numeric precision. It only affects the
>> statistic
>> > > model used to compute the p-value. When `exact=TRUE` the true
>> > distribution
>> > > of the statistic will be used. Otherwise, a normal approximation will
>> be
>> > > used.
>> > >
>> > > I think the documentation needs to be improved here, you can compute
>> the
>> > > exact p-value *only* when you do not have any ties in your data. If
>> you
>> > > have ties in your data you will get the p-value from the normal
>> > > approximation no matter what value you put in `exact`. This behavior
>> > should
>> > > be documented or a warning should be given when `exact=TRUE` and ties
>> > > present.
>> > >
>> > > FYI, if the exact p-value is required, `pwilcox` function will be
>> used to
>> > > compute the p-value. There are no details on how it computes the
>> pvalue
>> > but
>> > > its C code seems to compute the probability table, so I assume it
>> > computes
>> > > the exact p-value from the true distribution of the statistic, not a
>> > > permutation or MC p-value.
>> >
>> >
>> >        My example shows that it does NOT use Monte Carlo, because
>> > otherwise it uses some distribution.  I believe the term "exact" means
>> > that it uses the permutation distribution, though I could be mistaken.
>> > If it's NOT a permutation distribution, I don't know what it is.
>> >
>> >
>> >        Spencer
>> > >
>> > > Best,
>> > > Jiefei
>> > >
>> > >
>> > >
>> > > On Fri, Mar 19, 2021 at 10:01 PM Jiefei Wang <[hidden email]>
>> wrote:
>> > >
>> > >> Hey,
>> > >>
>> > >> I just want to point out that the word "exact" has two meanings. It
>> can
>> > >> mean the numerically accurate p-value as Bogdan asked in his first
>> > email,
>> > >> or it could mean the p-value calculated from the exact distribution
>> of
>> > the
>> > >> statistic(In this case, U stat). These two are actually not related,
>> > even
>> > >> though they all called "exact".
>> > >>
>> > >> Best,
>> > >> Jiefei
>> > >>
>> > >> On Fri, Mar 19, 2021 at 9:31 PM Spencer Graves <
>> > >> [hidden email]> wrote:
>> > >>
>> > >>>
>> > >>> On 2021-3-19 12:54 AM, Bogdan Tanasa wrote:
>> > >>>> thanks a lot, Vivek ! in other words, assuming that we work with
>> 1000
>> > >>> data
>> > >>>> points,
>> > >>>>
>> > >>>> shall we use EXACT = TRUE, it uses the normal approximation,
>> > >>>>
>> > >>>> while if EXACT=FALSE (for these large samples), it does not ?
>> > >>>
>> > >>>         As David Winsemius noted, the documentation is not clear.
>> > >>> Consider the following:
>> > >>>
>> > >>>> set.seed(1)  > x <- rnorm(100) > y <- rnorm(100, 2) > >
>> wilcox.test(x,
>> > >>> y)$p.value
>> > >>> [1] 1.172189e-25 > wilcox.test(x, y)$p.value [1] 1.172189e-25 > >
>> > >>> wilcox.test(x, y, EXACT=TRUE)$p.value [1] 1.172189e-25 >
>> wilcox.test(x,
>> > >>> y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>> > >>> exact=TRUE)$p.value [1] 4.123875e-32 > wilcox.test(x, y,
>> > >>> exact=TRUE)$p.value [1] 4.123875e-32 > > wilcox.test(x, y,
>> > >>> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>> > >>> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>> > >>> exact=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>> > >>> exact=FALSE)$p.value [1] 1.172189e-25 > We get two values here:
>> > >>> 1.172189e-25 and 4.123875e-32. The first one, I think, is the normal
>> > >>> approximation, which is the same as exact=FALSE. I think that with
>> > >>> exact=FALSE, you get a permutation distribution, though I'm not
>> sure.
>> > >>> You might try looking at "wilcox_test in package coin for exact,
>> > >>> asymptotic and Monte Carlo conditional p-values, including in the
>> > >>> presence of ties" to see if it is clearer. NOTE: R is case
>> sensitive,
>> > so
>> > >>> "EXACT" is a different variable from "exact". It is interpreted as
>> an
>> > >>> optional argument, which is not recognized and therefore ignored in
>> > this
>> > >>> context.
>> > >>>            Hope this helps.
>> > >>>            Spencer
>> > >>>
>> > >>>
>> > >>>> On Thu, Mar 18, 2021 at 10:47 PM Vivek Das <[hidden email]>
>> > wrote:
>> > >>>>
>> > >>>>> Hi Bogdan,
>> > >>>>>
>> > >>>>> You can also get the information from the link of the Wilcox.test
>> > >>> function
>> > >>>>> page.
>> > >>>>>
>> > >>>>> “By default (if exact is not specified), an exact p-value is
>> computed
>> > >>> if
>> > >>>>> the samples contain less than 50 finite values and there are no
>> ties.
>> > >>>>> Otherwise, a normal approximation is used.”
>> > >>>>>
>> > >>>>> For more:
>> > >>>>>
>> > >>>>>
>> > >>>
>> >
>> https://stat.ethz.ch/R-manual/R-devel/library/stats/html/wilcox.test.html
>> > >>>>> Hope this helps!
>> > >>>>>
>> > >>>>> Best,
>> > >>>>>
>> > >>>>> VD
>> > >>>>>
>> > >>>>>
>> > >>>>> On Thu, Mar 18, 2021 at 10:36 PM Bogdan Tanasa <[hidden email]>
>> > >>> wrote:
>> > >>>>>> Dear Peter, thanks a lot. yes, we can see a very precise p-value,
>> > and
>> > >>> that
>> > >>>>>> was the request from the journal.
>> > >>>>>>
>> > >>>>>> if I may ask another question please : what is the meaning of
>> > >>> "exact=TRUE"
>> > >>>>>> or "exact=FALSE" in wilcox.test ?
>> > >>>>>>
>> > >>>>>> i can see that the "numerically precise" p-values are different.
>> > >>> thanks a
>> > >>>>>> lot !
>> > >>>>>>
>> > >>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>> > >>>>>> tst$p.value
>> > >>>>>> [1] 8.535524e-25
>> > >>>>>>
>> > >>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=FALSE)
>> > >>>>>> tst$p.value
>> > >>>>>> [1] 3.448211e-25
>> > >>>>>>
>> > >>>>>> On Thu, Mar 18, 2021 at 10:15 PM Peter Langfelder <
>> > >>>>>> [hidden email]> wrote:
>> > >>>>>>
>> > >>>>>>> I thinnk the answer is much simpler. The print method for
>> > hypothesis
>> > >>>>>>> tests (class htest) truncates the p-values. In the above
>> example,
>> > >>>>>>> instead of using
>> > >>>>>>>
>> > >>>>>>> wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>> > >>>>>>>
>> > >>>>>>> and copying the output, just print the p-value:
>> > >>>>>>>
>> > >>>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>> > >>>>>>> tst$p.value
>> > >>>>>>>
>> > >>>>>>> [1] 2.988368e-32
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>> I think this value is what the journal asks for.
>> > >>>>>>>
>> > >>>>>>> HTH,
>> > >>>>>>>
>> > >>>>>>> Peter
>> > >>>>>>>
>> > >>>>>>> On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves
>> > >>>>>>> <[hidden email]> wrote:
>> > >>>>>>>>          I would push back on that from two perspectives:
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>                1.  I would study exactly what the journal said
>> > very
>> > >>>>>>>> carefully.  If they mandated "wilcox.test", that function has
>> an
>> > >>>>>>>> argument called "exact".  If that's what they are asking, then
>> > using
>> > >>>>>>>> that argument gives the exact p-value, e.g.:
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>    > wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>> > >>>>>>>>
>> > >>>>>>>>            Wilcoxon rank sum exact test
>> > >>>>>>>>
>> > >>>>>>>> data:  rnorm(100) and rnorm(100, 2)
>> > >>>>>>>> W = 691, p-value < 2.2e-16
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>                2.  If that's NOT what they are asking, then I'm
>> > not
>> > >>>>>>>> convinced what they are asking makes sense:  There is is no
>> such
>> > >>> thing
>> > >>>>>>>> as an "exact p value" except to the extent that certain
>> > assumptions
>> > >>>>>>>> hold, and all models are wrong (but some are useful), as George
>> > Box
>> > >>>>>>>> famously said years ago.[1]  Truth only exists in mathematics,
>> and
>> > >>>>>>>> that's because it's a fiction to start with ;-)
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>          Hope this helps.
>> > >>>>>>>>          Spencer Graves
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> [1]
>> > >>>>>>>> https://en.wikipedia.org/wiki/All_models_are_wrong
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
>> > >>>>>>>>>     <
>> > >>>
>> > https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16
>> > >>>>>>>>> Dear all,
>> > >>>>>>>>>
>> > >>>>>>>>> i would appreciate having your advice on the following please
>> :
>> > >>>>>>>>>
>> > >>>>>>>>> in R, the wilcox.test() provides "a p-value < 2.2e-16", when
>> we
>> > >>>>>> compare
>> > >>>>>>>>> sets of 1000 genes expression (in the genomics field).
>> > >>>>>>>>>
>> > >>>>>>>>> however, the journal asks us to provide the exact p value ...
>> > >>>>>>>>>
>> > >>>>>>>>> would it be legitimate to write : "p-value = 0" ? thanks a
>> lot,
>> > >>>>>>>>>
>> > >>>>>>>>> -- bogdan
>> > >>>>>>>>>
>> > >>>>>>>>>         [[alternative HTML version deleted]]
>> > >>>>>>>>>
>> > >>>>>>>>> ______________________________________________
>> > >>>>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more,
>> > see
>> > >>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>> > >>>>>>>>> PLEASE do read the posting guide
>> > >>>>>>> http://www.R-project.org/posting-guide.html
>> > >>>>>>>>> and provide commented, minimal, self-contained, reproducible
>> > code.
>> > >>>>>>>> ______________________________________________
>> > >>>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more,
>> see
>> > >>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>> > >>>>>>>> PLEASE do read the posting guide
>> > >>>>>>> http://www.R-project.org/posting-guide.html
>> > >>>>>>>> and provide commented, minimal, self-contained, reproducible
>> code.
>> > >>>>>>           [[alternative HTML version deleted]]
>> > >>>>>>
>> > >>>>>> ______________________________________________
>> > >>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more,
>> see
>> > >>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>> > >>>>>> PLEASE do read the posting guide
>> > >>>>>> http://www.R-project.org/posting-guide.html
>> > >>>>>> and provide commented, minimal, self-contained, reproducible
>> code.
>> > >>>>>>
>> > >>>>> --
>> > >>>>> ----------------------------------------------------------
>> > >>>>>
>> > >>>>> Vivek Das, PhD
>> > >>>>>
>> > >>>>        [[alternative HTML version deleted]]
>> > >>>>
>> > >>>> ______________________________________________
>> > >>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> > >>>> https://stat.ethz.ch/mailman/listinfo/r-help
>> > >>>> PLEASE do read the posting guide
>> > >>> http://www.R-project.org/posting-guide.html
>> > >>>> and provide commented, minimal, self-contained, reproducible code.
>> > >>>
>> > >>>          [[alternative HTML version deleted]]
>> > >>>
>> > >>> ______________________________________________
>> > >>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> > >>> https://stat.ethz.ch/mailman/listinfo/r-help
>> > >>> PLEASE do read the posting guide
>> > >>> http://www.R-project.org/posting-guide.html
>> > >>> and provide commented, minimal, self-contained, reproducible code.
>> > >>>
>> >
>> >
>>
>>         [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: about a p-value < 2.2e-16

Kevin Thorpe
I have to ask since. Are you sure the journal simply means by exact p-value that they don’t want to see a p-value given as < 0.0001, for example, and simply want the actual number?

I cannot imagine they really meant exact as in the p-value from some exact distribution.

--
Kevin E. Thorpe
Head of Biostatistics,  Applied Health Research Centre (AHRC)
Li Ka Shing Knowledge Institute of St. Michael's
Assistant Professor, Dalla Lana School of Public Health
University of Toronto
email: [hidden email]  Tel: 416.864.5776  Fax: 416.864.3016

> On Mar 19, 2021, at 1:22 PM, Bogdan Tanasa <[hidden email]> wrote:
>
> EXTERNAL EMAIL:
>
> Dear all, thank you all for comments and help.
>
> as far as i can see, shall we have samples of 1000 records, only
> "exact=FALSE" allows the code to run:
>
> wilcox.test(rnorm(1000), rnorm(1000, 2), exact=FALSE)$p.value
> [1] 7.304863e-231
>
> shall i use "exact=TRUE", it runs out of memory on my 64GB RAM PC :
>
> wilcox.test(rnorm(1000), rnorm(1000, 2), exact=TRUE)$p.value
> (the job is terminated by OS)
>
> shall you have any other suggestions, please let me know. thanks a lot !
>
> On Fri, Mar 19, 2021 at 9:05 AM Bert Gunter <[hidden email]> wrote:
>
>> I **believe** -- if my old memory still serves-- that the "exact"
>> specification uses a home grown version of the algorithm to calculate
>> exact,  or close approximations to the exact, permutation distribution
>> originally developed by Cyrus Mehta, founder of StatXact software.  Of
>> course, examining the C code source would determine this, but I don't care
>> to attempt this.
>>
>> If this is (no longer?) correct, please point this out.
>>
>> Best,
>>
>> Bert Gunter
>>
>> "The trouble with having an open mind is that people keep coming along and
>> sticking things into it."
>> -- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )
>>
>>
>> On Fri, Mar 19, 2021 at 8:42 AM Jiefei Wang <[hidden email]> wrote:
>>
>>> Hi Spencer,
>>>
>>> Thanks for your test results, I do not know the answer as I haven't
>>> used wilcox.test for many years. I do not know if it is possible to
>>> compute
>>> the exact distribution of the Wilcoxon rank sum statistic, but I think it
>>> is very likely, as the document of `Wilcoxon` says:
>>>
>>> This distribution is obtained as follows. Let x and y be two random,
>>> independent samples of size m and n. Then the Wilcoxon rank sum statistic
>>> is the number of all pairs (x[i], y[j]) for which y[j] is not greater than
>>> x[i]. This statistic takes values between 0 and m * n, and its mean and
>>> variance are m * n / 2 and m * n * (m + n + 1) / 12, respectively.
>>>
>>> As a nice feature of the non-parametric statistic, it is usually
>>> distribution-free so you can pick any distribution you like to compute the
>>> same statistic. I wonder if this is the case, but I might be wrong.
>>>
>>> Cheers,
>>> Jiefei
>>>
>>>
>>> On Fri, Mar 19, 2021 at 10:57 PM Spencer Graves <
>>> [hidden email]> wrote:
>>>
>>>>
>>>>
>>>> On 2021-3-19 9:52 AM, Jiefei Wang wrote:
>>>>> After digging into the R source, it turns out that the argument
>>> `exact`
>>>> has
>>>>> nothing to do with the numeric precision. It only affects the
>>> statistic
>>>>> model used to compute the p-value. When `exact=TRUE` the true
>>>> distribution
>>>>> of the statistic will be used. Otherwise, a normal approximation will
>>> be
>>>>> used.
>>>>>
>>>>> I think the documentation needs to be improved here, you can compute
>>> the
>>>>> exact p-value *only* when you do not have any ties in your data. If
>>> you
>>>>> have ties in your data you will get the p-value from the normal
>>>>> approximation no matter what value you put in `exact`. This behavior
>>>> should
>>>>> be documented or a warning should be given when `exact=TRUE` and ties
>>>>> present.
>>>>>
>>>>> FYI, if the exact p-value is required, `pwilcox` function will be
>>> used to
>>>>> compute the p-value. There are no details on how it computes the
>>> pvalue
>>>> but
>>>>> its C code seems to compute the probability table, so I assume it
>>>> computes
>>>>> the exact p-value from the true distribution of the statistic, not a
>>>>> permutation or MC p-value.
>>>>
>>>>
>>>>       My example shows that it does NOT use Monte Carlo, because
>>>> otherwise it uses some distribution.  I believe the term "exact" means
>>>> that it uses the permutation distribution, though I could be mistaken.
>>>> If it's NOT a permutation distribution, I don't know what it is.
>>>>
>>>>
>>>>       Spencer
>>>>>
>>>>> Best,
>>>>> Jiefei
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Mar 19, 2021 at 10:01 PM Jiefei Wang <[hidden email]>
>>> wrote:
>>>>>
>>>>>> Hey,
>>>>>>
>>>>>> I just want to point out that the word "exact" has two meanings. It
>>> can
>>>>>> mean the numerically accurate p-value as Bogdan asked in his first
>>>> email,
>>>>>> or it could mean the p-value calculated from the exact distribution
>>> of
>>>> the
>>>>>> statistic(In this case, U stat). These two are actually not related,
>>>> even
>>>>>> though they all called "exact".
>>>>>>
>>>>>> Best,
>>>>>> Jiefei
>>>>>>
>>>>>> On Fri, Mar 19, 2021 at 9:31 PM Spencer Graves <
>>>>>> [hidden email]> wrote:
>>>>>>
>>>>>>>
>>>>>>> On 2021-3-19 12:54 AM, Bogdan Tanasa wrote:
>>>>>>>> thanks a lot, Vivek ! in other words, assuming that we work with
>>> 1000
>>>>>>> data
>>>>>>>> points,
>>>>>>>>
>>>>>>>> shall we use EXACT = TRUE, it uses the normal approximation,
>>>>>>>>
>>>>>>>> while if EXACT=FALSE (for these large samples), it does not ?
>>>>>>>
>>>>>>>        As David Winsemius noted, the documentation is not clear.
>>>>>>> Consider the following:
>>>>>>>
>>>>>>>> set.seed(1)  > x <- rnorm(100) > y <- rnorm(100, 2) > >
>>> wilcox.test(x,
>>>>>>> y)$p.value
>>>>>>> [1] 1.172189e-25 > wilcox.test(x, y)$p.value [1] 1.172189e-25 > >
>>>>>>> wilcox.test(x, y, EXACT=TRUE)$p.value [1] 1.172189e-25 >
>>> wilcox.test(x,
>>>>>>> y, EXACT=TRUE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>>>>>>> exact=TRUE)$p.value [1] 4.123875e-32 > wilcox.test(x, y,
>>>>>>> exact=TRUE)$p.value [1] 4.123875e-32 > > wilcox.test(x, y,
>>>>>>> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>>>>>>> EXACT=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>>>>>>> exact=FALSE)$p.value [1] 1.172189e-25 > wilcox.test(x, y,
>>>>>>> exact=FALSE)$p.value [1] 1.172189e-25 > We get two values here:
>>>>>>> 1.172189e-25 and 4.123875e-32. The first one, I think, is the normal
>>>>>>> approximation, which is the same as exact=FALSE. I think that with
>>>>>>> exact=FALSE, you get a permutation distribution, though I'm not
>>> sure.
>>>>>>> You might try looking at "wilcox_test in package coin for exact,
>>>>>>> asymptotic and Monte Carlo conditional p-values, including in the
>>>>>>> presence of ties" to see if it is clearer. NOTE: R is case
>>> sensitive,
>>>> so
>>>>>>> "EXACT" is a different variable from "exact". It is interpreted as
>>> an
>>>>>>> optional argument, which is not recognized and therefore ignored in
>>>> this
>>>>>>> context.
>>>>>>>           Hope this helps.
>>>>>>>           Spencer
>>>>>>>
>>>>>>>
>>>>>>>> On Thu, Mar 18, 2021 at 10:47 PM Vivek Das <[hidden email]>
>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Bogdan,
>>>>>>>>>
>>>>>>>>> You can also get the information from the link of the Wilcox.test
>>>>>>> function
>>>>>>>>> page.
>>>>>>>>>
>>>>>>>>> “By default (if exact is not specified), an exact p-value is
>>> computed
>>>>>>> if
>>>>>>>>> the samples contain less than 50 finite values and there are no
>>> ties.
>>>>>>>>> Otherwise, a normal approximation is used.”
>>>>>>>>>
>>>>>>>>> For more:
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>
>>> https://stat.ethz.ch/R-manual/R-devel/library/stats/html/wilcox.test.html
>>>>>>>>> Hope this helps!
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>>
>>>>>>>>> VD
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 18, 2021 at 10:36 PM Bogdan Tanasa <[hidden email]>
>>>>>>> wrote:
>>>>>>>>>> Dear Peter, thanks a lot. yes, we can see a very precise p-value,
>>>> and
>>>>>>> that
>>>>>>>>>> was the request from the journal.
>>>>>>>>>>
>>>>>>>>>> if I may ask another question please : what is the meaning of
>>>>>>> "exact=TRUE"
>>>>>>>>>> or "exact=FALSE" in wilcox.test ?
>>>>>>>>>>
>>>>>>>>>> i can see that the "numerically precise" p-values are different.
>>>>>>> thanks a
>>>>>>>>>> lot !
>>>>>>>>>>
>>>>>>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>>>>>>>>> tst$p.value
>>>>>>>>>> [1] 8.535524e-25
>>>>>>>>>>
>>>>>>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=FALSE)
>>>>>>>>>> tst$p.value
>>>>>>>>>> [1] 3.448211e-25
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 18, 2021 at 10:15 PM Peter Langfelder <
>>>>>>>>>> [hidden email]> wrote:
>>>>>>>>>>
>>>>>>>>>>> I thinnk the answer is much simpler. The print method for
>>>> hypothesis
>>>>>>>>>>> tests (class htest) truncates the p-values. In the above
>>> example,
>>>>>>>>>>> instead of using
>>>>>>>>>>>
>>>>>>>>>>> wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>>>>>>>>>>
>>>>>>>>>>> and copying the output, just print the p-value:
>>>>>>>>>>>
>>>>>>>>>>> tst = wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>>>>>>>>>> tst$p.value
>>>>>>>>>>>
>>>>>>>>>>> [1] 2.988368e-32
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I think this value is what the journal asks for.
>>>>>>>>>>>
>>>>>>>>>>> HTH,
>>>>>>>>>>>
>>>>>>>>>>> Peter
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 18, 2021 at 10:05 PM Spencer Graves
>>>>>>>>>>> <[hidden email]> wrote:
>>>>>>>>>>>>         I would push back on that from two perspectives:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>               1.  I would study exactly what the journal said
>>>> very
>>>>>>>>>>>> carefully.  If they mandated "wilcox.test", that function has
>>> an
>>>>>>>>>>>> argument called "exact".  If that's what they are asking, then
>>>> using
>>>>>>>>>>>> that argument gives the exact p-value, e.g.:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> wilcox.test(rnorm(100), rnorm(100, 2), exact=TRUE)
>>>>>>>>>>>>
>>>>>>>>>>>>           Wilcoxon rank sum exact test
>>>>>>>>>>>>
>>>>>>>>>>>> data:  rnorm(100) and rnorm(100, 2)
>>>>>>>>>>>> W = 691, p-value < 2.2e-16
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>               2.  If that's NOT what they are asking, then I'm
>>>> not
>>>>>>>>>>>> convinced what they are asking makes sense:  There is is no
>>> such
>>>>>>> thing
>>>>>>>>>>>> as an "exact p value" except to the extent that certain
>>>> assumptions
>>>>>>>>>>>> hold, and all models are wrong (but some are useful), as George
>>>> Box
>>>>>>>>>>>> famously said years ago.[1]  Truth only exists in mathematics,
>>> and
>>>>>>>>>>>> that's because it's a fiction to start with ;-)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>         Hope this helps.
>>>>>>>>>>>>         Spencer Graves
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://en.wikipedia.org/wiki/All_models_are_wrong
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 2021-3-18 11:12 PM, Bogdan Tanasa wrote:
>>>>>>>>>>>>>    <
>>>>>>>
>>>> https://meta.stackexchange.com/questions/362285/about-a-p-value-2-2e-16
>>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> i would appreciate having your advice on the following please
>>> :
>>>>>>>>>>>>>
>>>>>>>>>>>>> in R, the wilcox.test() provides "a p-value < 2.2e-16", when
>>> we
>>>>>>>>>> compare
>>>>>>>>>>>>> sets of 1000 genes expression (in the genomics field).
>>>>>>>>>>>>>
>>>>>>>>>>>>> however, the journal asks us to provide the exact p value ...
>>>>>>>>>>>>>
>>>>>>>>>>>>> would it be legitimate to write : "p-value = 0" ? thanks a
>>> lot,
>>>>>>>>>>>>>
>>>>>>>>>>>>> -- bogdan
>>>>>>>>>>>>>
>>>>>>>>>>>>>        [[alternative HTML version deleted]]
>>>>>>>>>>>>>
>>>>>>>>>>>>> ______________________________________________
>>>>>>>>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more,
>>>> see
>>>>>>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>>>>>>>>>>> PLEASE do read the posting guide
>>>>>>>>>>> http://www.R-project.org/posting-guide.html
>>>>>>>>>>>>> and provide commented, minimal, self-contained, reproducible
>>>> code.
>>>>>>>>>>>> ______________________________________________
>>>>>>>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more,
>>> see
>>>>>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>>>>>>>>>> PLEASE do read the posting guide
>>>>>>>>>>> http://www.R-project.org/posting-guide.html
>>>>>>>>>>>> and provide commented, minimal, self-contained, reproducible
>>> code.
>>>>>>>>>>          [[alternative HTML version deleted]]
>>>>>>>>>>
>>>>>>>>>> ______________________________________________
>>>>>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more,
>>> see
>>>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>>>>>>>> PLEASE do read the posting guide
>>>>>>>>>> http://www.R-project.org/posting-guide.html
>>>>>>>>>> and provide commented, minimal, self-contained, reproducible
>>> code.
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>
>>>>>>>>> Vivek Das, PhD
>>>>>>>>>
>>>>>>>>       [[alternative HTML version deleted]]
>>>>>>>>
>>>>>>>> ______________________________________________
>>>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>>>>>> PLEASE do read the posting guide
>>>>>>> http://www.R-project.org/posting-guide.html
>>>>>>>> and provide commented, minimal, self-contained, reproducible code.
>>>>>>>
>>>>>>>         [[alternative HTML version deleted]]
>>>>>>>
>>>>>>> ______________________________________________
>>>>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>>>>> PLEASE do read the posting guide
>>>>>>> http://www.R-project.org/posting-guide.html
>>>>>>> and provide commented, minimal, self-contained, reproducible code.
>>>>>>>
>>>>
>>>>
>>>
>>>        [[alternative HTML version deleted]]
>>>
>>> ______________________________________________
>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>
>
>        [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
12