Randomization Test

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Randomization Test

Ogbos
Dear Contributors,

I conducting epoch analysis. I tried to test the significance of my
result using randomization test.

Since I have 71 events, I randomly selected another 71 events, making
sure that none of the dates in the random events corresponds with the
ones in the real event.

Following the code I found here
(https://www.uvm.edu/~dhowell/StatPages/R/RandomizationTestsWithR/Random2Sample/TwoIndependentSamplesR.html),
I combined these two data set and used them to generate another 5000
events. I then plotted the graph of the mean differences for the 5000
randomly generated events. On the graph, I indicated the region of the
mean difference between the real 71 epoch and the randomly selected 71
epoch.

Since the two tail test shows that the mean difference falls at the
extreme of the randomly selected events, I concluded that my result is
statistically significant.



I am attaching the graph to assistance you in you suggestions.

I can attach both my code and the real and randomly generated events
if you ask for it.

My request is that you help me to understand if I am on the right
track or no. This is the first time I am doing this and except the
experts decide, I am not quite sure whether I am right or not.

Many thanks for your kind concern.

Best
Ogbos
______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Randomization Test

Meyners, Michael
Ogbos,

You do not seem to have received a reply over the list yet, which might be due to the fact that this seems rather a stats than an R question. Neither got your attachment (Figure) through - see posting guide.

I'm not familiar with epoch analysis, so not sure what exactly you are doing / trying to achieve, but some general thoughts:

* You do NOT want to restrict your re-randomizations in a way that "none of the dates corresponds with the ones in the real event" - actually, as a general principle, the true data must be an admissible re-randomization as well. You seem to have excluded that (and a lot of other randomizations at the same time which might have occurred, i.e. dates 1 and 2 reversed but all others the same), thereby rendering the test invalid. Any restrictions you have on your re-randomizations must've applied to the original randomization as well.
* If you have rather observational data (which I suspect, but not sure), Edgington & Onghena (2007) would rather refer to this as a permutation test - the difference being that you have to make strong assumptions (similar to parametric tests) on the nature of the data, which are designed-in to be true for randomization tests. It might be a merely linguistic discrimination, but it is important to note which assumptions have to be (implicitly) made.
* I'm not sure what you mean by "mean differences" of the events - is that two groups you are comparing? If so, that seems reasonable, but just make sure the test statistic you use is reasonable and sensitive against the alternatives you are mostly interested in. The randomization/permutation test will never proof that, e.g., means are significantly different, but only that there is SOME difference. By selecting the appropriate test statistic, you can influence what will pop up more easily and what not, but you can never be sure (unless you make strong assumptions about everything else, like in many parametric tests).
* For any test statistic, you would then determine the proportion of its values among the 5000 samples where it is as large or larger than the one observed (or as small or smaller, or either, depending on the nature of the test statistic and whether you aim for a one- or a two-sided test). That is your p value. If small enough, conclude significance. At least conceptually important: The observed test statistic is always part of the re-randomization (i.e. your 5000) - so you truly only do 4999 plus the one you observed. Otherwise the test may be more or less liberal. Your p value is hence no smaller than 1/n, where n is the total number of samples you looked at (including the observed one), a p value of 0 is not possible in randomization tests (nor in other tests, of course).

I hope this is helpful, but you will need to go through these and refer to your own setup to check whether you adhered to the principles or not, which is impossible for me to judge based on the information provided (and I won't be able to look at excessive code to check either).

Michael

> -----Original Message-----
> From: R-help <[hidden email]> On Behalf Of Ogbos Okike
> Sent: Montag, 28. Januar 2019 19:42
> To: r-help <[hidden email]>
> Subject: [R] Randomization Test
>
> Dear Contributors,
>
> I conducting epoch analysis. I tried to test the significance of my result using
> randomization test.
>
> Since I have 71 events, I randomly selected another 71 events, making sure
> that none of the dates in the random events corresponds with the ones in
> the real event.
>
> Following the code I found here
> (https://www.uvm.edu/~dhowell/StatPages/R/RandomizationTestsWithR/R
> andom2Sample/TwoIndependentSamplesR.html),
> I combined these two data set and used them to generate another 5000
> events. I then plotted the graph of the mean differences for the 5000
> randomly generated events. On the graph, I indicated the region of the
> mean difference between the real 71 epoch and the randomly selected 71
> epoch.
>
> Since the two tail test shows that the mean difference falls at the extreme of
> the randomly selected events, I concluded that my result is statistically
> significant.
>
>
>
> I am attaching the graph to assistance you in you suggestions.
>
> I can attach both my code and the real and randomly generated events if you
> ask for it.
>
> My request is that you help me to understand if I am on the right track or no.
> This is the first time I am doing this and except the experts decide, I am not
> quite sure whether I am right or not.
>
> Many thanks for your kind concern.
>
> Best
> Ogbos
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-
> guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Randomization Test

Ogbos
Dear Michael,
This is great! Thank you.

I have not really got any response other than yours.

I have long before now included what I have in a paper submitted to a journal.

I am awaiting the feedback of the reviewer. I will compare the
comments with your input here and determine the corrections to make
and probably return to the list for additional help.

Best wishes
Ogbos

On Fri, Feb 8, 2019 at 4:31 PM Meyners, Michael <[hidden email]> wrote:

>
> Ogbos,
>
> You do not seem to have received a reply over the list yet, which might be due to the fact that this seems rather a stats than an R question. Neither got your attachment (Figure) through - see posting guide.
>
> I'm not familiar with epoch analysis, so not sure what exactly you are doing / trying to achieve, but some general thoughts:
>
> * You do NOT want to restrict your re-randomizations in a way that "none of the dates corresponds with the ones in the real event" - actually, as a general principle, the true data must be an admissible re-randomization as well. You seem to have excluded that (and a lot of other randomizations at the same time which might have occurred, i.e. dates 1 and 2 reversed but all others the same), thereby rendering the test invalid. Any restrictions you have on your re-randomizations must've applied to the original randomization as well.
> * If you have rather observational data (which I suspect, but not sure), Edgington & Onghena (2007) would rather refer to this as a permutation test - the difference being that you have to make strong assumptions (similar to parametric tests) on the nature of the data, which are designed-in to be true for randomization tests. It might be a merely linguistic discrimination, but it is important to note which assumptions have to be (implicitly) made.
> * I'm not sure what you mean by "mean differences" of the events - is that two groups you are comparing? If so, that seems reasonable, but just make sure the test statistic you use is reasonable and sensitive against the alternatives you are mostly interested in. The randomization/permutation test will never proof that, e.g., means are significantly different, but only that there is SOME difference. By selecting the appropriate test statistic, you can influence what will pop up more easily and what not, but you can never be sure (unless you make strong assumptions about everything else, like in many parametric tests).
> * For any test statistic, you would then determine the proportion of its values among the 5000 samples where it is as large or larger than the one observed (or as small or smaller, or either, depending on the nature of the test statistic and whether you aim for a one- or a two-sided test). That is your p value. If small enough, conclude significance. At least conceptually important: The observed test statistic is always part of the re-randomization (i.e. your 5000) - so you truly only do 4999 plus the one you observed. Otherwise the test may be more or less liberal. Your p value is hence no smaller than 1/n, where n is the total number of samples you looked at (including the observed one), a p value of 0 is not possible in randomization tests (nor in other tests, of course).
>
> I hope this is helpful, but you will need to go through these and refer to your own setup to check whether you adhered to the principles or not, which is impossible for me to judge based on the information provided (and I won't be able to look at excessive code to check either).
>
> Michael
>
> > -----Original Message-----
> > From: R-help <[hidden email]> On Behalf Of Ogbos Okike
> > Sent: Montag, 28. Januar 2019 19:42
> > To: r-help <[hidden email]>
> > Subject: [R] Randomization Test
> >
> > Dear Contributors,
> >
> > I conducting epoch analysis. I tried to test the significance of my result using
> > randomization test.
> >
> > Since I have 71 events, I randomly selected another 71 events, making sure
> > that none of the dates in the random events corresponds with the ones in
> > the real event.
> >
> > Following the code I found here
> > (https://www.uvm.edu/~dhowell/StatPages/R/RandomizationTestsWithR/R
> > andom2Sample/TwoIndependentSamplesR.html),
> > I combined these two data set and used them to generate another 5000
> > events. I then plotted the graph of the mean differences for the 5000
> > randomly generated events. On the graph, I indicated the region of the
> > mean difference between the real 71 epoch and the randomly selected 71
> > epoch.
> >
> > Since the two tail test shows that the mean difference falls at the extreme of
> > the randomly selected events, I concluded that my result is statistically
> > significant.
> >
> >
> >
> > I am attaching the graph to assistance you in you suggestions.
> >
> > I can attach both my code and the real and randomly generated events if you
> > ask for it.
> >
> > My request is that you help me to understand if I am on the right track or no.
> > This is the first time I am doing this and except the experts decide, I am not
> > quite sure whether I am right or not.
> >
> > Many thanks for your kind concern.
> >
> > Best
> > Ogbos
> > ______________________________________________
> > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide http://www.R-project.org/posting-
> > guide.html
> > and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Randomization Test

Ogbos
Dear Kind List,

I am still battling with this. I have, however, made some progress
with the suggestions of Micheal and others. At least, I have a better
picture of what I want to do now as I will attempt a detailed
description here.

I am aware I should show you just a small part of my code and data.
But when I copied out a small portion and run to see what you get when
I send that,  I was not satisfied with the signal displayed. The epoch
analysis averages data and is quite sensitive to leveraging,
especially if a small sample is used.

So please permit/exercise patience  me to display the series of epoch
that give the averaged valued used. You can just run the code and see
the signal of interest. Here is the code and the data:

dta <- read.table( text ="n CR
 -5 8969
-4 8932
-3 8929
-2 8916
-1 8807
0 8449
1 8484
2 8148
3 8282
4 8305
5 8380
6 8530
7 8642
8 8780
9 8890
10 8962
-5 8929
-4 8916
-3 8807
-2 8449
-1 8484
0 8148
1 8282
2 8305
3 8380
4 8530
5 8642
6 8780
7 8890
8 8962
9 8949
10 8974
-5 8744
-4 8786
-3 8828
-2 8807
-1 8716
0 8520
1 8634
2 8640
3 8636
4 8658
5 8699
6 8682
7 8621
8 8626
9 8660
10 8737
-5 8592
-4 8612
-3 8628
-2 8589
-1 8318
0 8264
1 8294
2 8410
3 8442
4 8416
5 8389
6 8412
7 8453
8 8563
9 8581
10 8613
-5 8264
-4 8294
-3 8410
-2 8442
-1 8416
0 8389
1 8412
2 8453
3 8563
4 8581
5 8613
6 8647
7 8613
8 8508
9 7829
10 7499
-5 8613
-4 8647
-3 8613
-2 8508
-1 7829
0 7499
1 8213
2 7993
3 7821
4 8316
5 8460
6 8533
7 8584
8 8586
9 8567
10 8573
-5 8508
-4 7829
-3 7499
-2 8213
-1 7993
0 7821
1 8316
2 8460
3 8533
4 8584
5 8586
6 8567
7 8573
8 8617
9 8591
10 8661
-5 8851
-4 8893
-3 8858
-2 8803
-1 8790
0 8468
1 8545
2 8570
3 8568
4 8624
5 8669
6 8236
7 8190
8 8313
9 8389
10 8421
-5 8803
-4 8790
-3 8468
-2 8545
-1 8570
0 8568
1 8624
2 8669
3 8236
4 8190
5 8313
6 8389
7 8421
8 8468
9 8537
10 8580
-5 8570
-4 8568
-3 8624
-2 8669
-1 8236
0 8190
1 8313
2 8389
3 8421
4 8468
5 8537
6 8580
7 8605
8 8646
9 8690
10 8770
-5 8690
-4 8770
-3 8799
-2 8821
-1 8666
0 8539
1 8633
2 8617
3 8651
4 8693
5 8715
6 8738
7 8716
8 8677
9 8680
10 8700
-5 8756
-4 8632
-3 8662
-2 8596
-1 8552
0 8502
1 8633
2 8702
3 8745
4 8730
5 8708
6 8817
7 8724
8 8688
9 8693
10 8746
-5 8926
-4 8888
-3 8798
-2 8651
-1 8678
0 8578
1 8593
2 8598
3 8526
4 8181
5 8204
6 8373
7 8599
8 8773
9 8784
10 8746
-5 8678
-4 8578
-3 8593
-2 8598
-1 8526
0 8181
1 8204
2 8373
3 8599
4 8773
5 8784
6 8746
7 8747
8 8757
9 8749
10 8767
-5 8757
-4 8749
-3 8767
-2 8754
-1 8695
0 8631
1 8661
2 8653
3 8588
4 8562
5 8613
6 8595
7 8498
8 8404
9 8507
10 8599
-5 8695
-4 8631
-3 8661
-2 8653
-1 8588
0 8562
1 8613
2 8595
3 8498
4 8404
5 8507
6 8599
7 8592
8 8600
9 8637
10 8635
-5 8588
-4 8562
-3 8613
-2 8595
-1 8498
0 8404
1 8507
2 8599
3 8592
4 8600
5 8637
6 8635
7 8632
8 8674
9 8644
10 8687
-5 8595
-4 8498
-3 8404
-2 8507
-1 8599
0 8592
1 8600
2 8637
3 8635
4 8632
5 8674
6 8644
7 8687
8 8721
9 8747
10 8748
-5 8599
-4 8592
-3 8600
-2 8637
-1 8635
0 8632
1 8674
2 8644
3 8687
4 8721
5 8747
6 8748
7 8739
8 8763
9 8792
10 8558
-5 8600
-4 8637
-3 8635
-2 8632
-1 8674
0 8644
1 8687
2 8721
3 8747
4 8748
5 8739
6 8763
7 8792
8 8558
9 8442
10 8555
-5 8748
-4 8739
-3 8763
-2 8792
-1 8558
0 8442
1 8555
2 8622
3 8634
4 8698
5 8732
6 8713
7 8732
8 8681
9 8615
10 8624
-5 8698
-4 8732
-3 8713
-2 8732
-1 8681
0 8615
1 8624
2 8649
3 8656
4 8678
5 8723
6 8693
7 8548
8 7803
9 7801
10 7724
-5 8723
-4 8693
-3 8548
-2 7803
-1 7801
0 7724
1 7910
2 7829
3 7995
4 8156
5 8307
6 8377
7 8465
8 8506
9 8516
10 8536
-5 8548
-4 7803
-3 7801
-2 7724
-1 7910
0 7829
1 7995
2 8156
3 8307
4 8377
5 8465
6 8506
7 8516
8 8536
9 8574
10 8623
-5 8821
-4 8856
-3 8798
-2 8772
-1 8705
0 8682
1 8691
2 8720
3 8727
4 8789
5 8821
6 8811
7 8841
8 8849
9 8849
10 8860
-5 8835
-4 8829
-3 8826
-2 8799
-1 8775
0 8756
1 8793
2 8814
3 8847
4 8838
5 8833
6 8841
7 8847
8 8903
9 8933
10 8918
-5 8890
-4 8875
-3 8874
-2 8865
-1 8891
0 8839
1 8853
2 8888
3 8884
4 8890
5 8889
6 8839
7 8879
8 8908
9 8924
10 8882
-5 8853
-4 8888
-3 8884
-2 8890
-1 8889
0 8839
1 8879
2 8908
3 8924
4 8882
5 8910
6 8903
7 8859
8 8858
9 8863
10 8847
-5 8924
-4 8882
-3 8910
-2 8903
-1 8859
0 8858
1 8863
2 8847
3 8883
4 8869
5 8878
6 8897
7 8922
8 8895
9 8858
10 8858
-5 8910
-4 8903
-3 8859
-2 8858
-1 8863
0 8847
1 8883
2 8869
3 8878
4 8897
5 8922
6 8895
7 8858
8 8858
9 8736
10 8905
-5 8859
-4 8858
-3 8863
-2 8847
-1 8883
0 8869
1 8878
2 8897
3 8922
4 8895
5 8858
6 8858
7 8736
8 8905
9 8935
10 8974
-5 8897
-4 8922
-3 8895
-2 8858
-1 8858
0 8736
1 8905
2 8935
3 8974
4 8946
5 8952
6 9010
7 8980
8 8976
9 8970
10 8961
-5 9376
-4 9336
-3 9311
-2 9287
-1 9221
0 9087
1 9132
2 9175
3 9166
4 9240
5 9264
6 9271
7 9319
8 9324
9 9333
10 9351
-5 9287
-4 9221
-3 9087
-2 9132
-1 9175
0 9166
1 9240
2 9264
3 9271
4 9319
5 9324
6 9333
7 9351
8 9362
9 9385
10 9354
-5 9407
-4 9414
-3 9354
-2 9298
-1 9319
0 9147
1 9178
2 9196
3 9258
4 9303
5 9369
6 9382
7 9375
8 9389
9 9376
10 9264
-5 9386
-4 9396
-3 9424
-2 9391
-1 9284
0 9267
1 9278
2 9318
3 9334
4 9275
5 9306
6 9308
7 9358
8 9335
9 9373
10 9379
-5 9284
-4 9267
-3 9278
-2 9318
-1 9334
0 9275
1 9306
2 9308
3 9358
4 9335
5 9373
6 9379
7 9355
8 9340
9 9327
10 9320
-5 9327
-4 9320
-3 9315
-2 9336
-1 9371
0 9259
1 9330
2 9355
3 9334
4 9353
5 9370
6 9394
7 9400
8 9318
9 9037
10 8994
-5 9394
-4 9400
-3 9318
-2 9037
-1 8994
0 8943
1 8964
2 8997
3 9158
4 8964
5 8564
6 8736
7 8818
8 8938
9 9034
10 9132
-5 8943
-4 8964
-3 8997
-2 9158
-1 8964
0 8564
1 8736
2 8818
3 8938
4 9034
5 9132
6 9167
7 9200
8 9257
9 9266
10 9306
-5 9338
-4 9354
-3 9372
-2 9338
-1 9308
0 9282
1 9324
2 9318
3 9342
4 9370
5 9331
6 9327
7 9338
8 9381
9 9394
10 9332
-5 9372
-4 9338
-3 9308
-2 9282
-1 9324
0 9318
1 9342
2 9370
3 9331
4 9327
5 9338
6 9381
7 9394
8 9332
9 9331
10 9293
-5 9338
-4 9381
-3 9394
-2 9332
-1 9331
0 9293
1 9309
2 9325
3 9406
4 9409
5 9413
6 9426
7 9440
8 9449
9 9512
10 9494
-5 9361
-4 9354
-3 9299
-2 9282
-1 9250
0 9242
1 9254
2 9321
3 9390
4 9414
5 9435
6 9437
7 9426
8 9398
9 9383
10 9354
-5 9365
-4 9421
-3 9416
-2 9355
-1 9338
0 9324
1 9325
2 9322
3 9319
4 9381
5 9315
6 9314
7 9359
8 9403
9 9419
10 9474
-5 9355
-4 9338
-3 9324
-2 9325
-1 9322
0 9319
1 9381
2 9315
3 9314
4 9359
5 9403
6 9419
7 9474
8 9525
9 9501
10 9447
-5 9325
-4 9322
-3 9319
-2 9381
-1 9315
0 9314
1 9359
2 9403
3 9419
4 9474
5 9525
6 9501
7 9447
8 9424
9 9396
10 9388
-5 9447
-4 9424
-3 9396
-2 9388
-1 9396
0 9346
1 9358
2 9353
3 9350
4 9378
5 9372
6 9354
7 9349
8 9392
9 9440
10 9467
-5 9388
-4 9396
-3 9346
-2 9358
-1 9353
0 9350
1 9378
2 9372
3 9354
4 9349
5 9392
6 9440
7 9467
8 9519
9 9550
10 9565
-5 9353
-4 9350
-3 9378
-2 9372
-1 9354
0 9349
1 9392
2 9440
3 9467
4 9519
5 9550
6 9565
7 9565
8 9497
9 9500
10 9472
-5 9522
-4 9529
-3 9492
-2 9432
-1 9382
0 9355
1 9361
2 9350
3 9382
4 9451
5 9491
6 9506
7 9529
8 9543
9 9556
10 9553
-5 9492
-4 9432
-3 9382
-2 9355
-1 9361
0 9350
1 9382
2 9451
3 9491
4 9506
5 9529
6 9543
7 9556
8 9553
9 9502
10 9470
-5 9551
-4 9505
-3 9389
-2 9406
-1 9377
0 9284
1 9365
2 9424
3 9412
4 9403
5 9384
6 9394
7 9404
8 9413
9 9407
10 9405
-5 9579
-4 9576
-3 9543
-2 9451
-1 9421
0 9361
1 9394
2 9400
3 9387
4 9366
5 9346
6 9360
7 9385
8 9435
9 9443
10 9430
-5 9361
-4 9394
-3 9400
-2 9387
-1 9366
0 9346
1 9360
2 9385
3 9435
4 9443
5 9430
6 9454
7 9531
8 9547
9 9581
10 9540
-5 9510
-4 9546
-3 9564
-2 9508
-1 9422
0 9369
1 9395
2 9438
3 9423
4 9392
5 9368
6 9366
7 9348
8 9340
9 9375
10 9391
-5 9423
-4 9392
-3 9368
-2 9366
-1 9348
0 9340
1 9375
2 9391
3 9466
4 9545
5 9574
6 9564
7 9527
8 9513
9 9494
10 9542
-5 9511
-4 9491
-3 9457
-2 9453
-1 9402
0 9382
1 9407
2 9437
3 9403
4 9404
5 9425
6 9486
7 9457
8 9451
9 9423
10 9401
-5 9425
-4 9486
-3 9457
-2 9451
-1 9423
0 9401
1 9429
2 9422
3 9431
4 9462
5 9475
6 9474
7 9487
8 9493
9 9495
10 9499
-5 9404
-4 9385
-3 9363
-2 9399
-1 9411
0 9355
1 9357
2 9363
3 9382
4 9387
5 9408
6 9429
7 9456
8 9487
9 9526
10 9487
-5 9493
-4 9439
-3 9400
-2 9378
-1 9371
0 9369
1 9374
2 9305
3 9298
4 9298
5 9325
6 9381
7 9477
8 9508
9 9496
10 9517
-5 9371
-4 9369
-3 9374
-2 9305
-1 9298
0 9298
1 9325
2 9381
3 9477
4 9508
5 9496
6 9517
7 9561
8 9570
9 9546
10 9544
-5 9510
-4 9506
-3 9530
-2 9441
-1 9427
0 9393
1 9420
2 9444
3 9468
4 9484
5 9525
6 9542
7 9557
8 9548
9 9550
10 9593
-5 9589
-4 9598
-3 9527
-2 9417
-1 9390
0 9374
1 9386
2 9407
3 9453
4 9447
5 9419
6 9386
7 9373
8 9364
9 9376
10 9389
-5 9453
-4 9447
-3 9419
-2 9386
-1 9373
0 9364
1 9376
2 9389
3 9376
4 9375
5 9370
6 9391
7 9458
8 9446
9 9456
10 9463
-5 9364
-4 9376
-3 9389
-2 9376
-1 9375
0 9370
1 9391
2 9458
3 9446
4 9456
5 9463
6 9500
7 9486
8 9474
9 9495
10 9531
-5 9491
-4 9441
-3 9388
-2 9380
-1 9369
0 9354
1 9367
2 9369
3 9341
4 9305
5 9308
6 9324
7 9385
8 9451
9 9496
10 9527
-5 9369
-4 9354
-3 9367
-2 9369
-1 9341
0 9305
1 9308
2 9324
3 9385
4 9451
5 9496
6 9527
7 9544
8 9543
9 9535
10 9536
-5 9586
-4 9583
-3 9572
-2 9533
-1 9454
0 9392
1 9420
2 9451
3 9475
4 9514
5 9561
6 9542
7 9502
8 9461
9 9468
10 9463
-5 9587
-4 9562
-3 9530
-2 9445
-1 9404
0 9395
1 9417
2 9449
3 9467
4 9470
5 9524
6 9512
7 9448
8 9398
9 9431
10 9467
-5 9467
-4 9470
-3 9524
-2 9512
-1 9448
0 9398
1 9431
2 9467
3 9490
4 9517
5 9526
6 9574
7 9573
8 9562
9 9563
10 9566
",header=TRUE)

 data<-matrix(c(dta$CR),ncol=71)
A<-matrix(rep(-5:10,71))
B<-matrix(data)

 oodf<-data.frame(A,B)
 a<--5:10
oodf<-data.frame(A,B)
library(plotrix)
std.error<-function(x) return(sd(x)/(sum(!is.na(x))))
oomean<-as.vector(by(oodf$B,oodf$A,mean))
oose<-as.vector(by(oodf$B,oodf$A,std.error))
plot(-5:10,oomean,type="l",ylim=c(8890,9100),
 )
A<-oomean-1.96*oose
 B<-oomean+1.96*oose
lines(a,A,col="red")
 lines(a,B,col="red")

 My Question:
I wish to conduct a randomization test of significance (90 and 99
percentile) of the reductions/decreases as displayed by the signal.

I am attempting using:
x<-sample(8890:9500,1136,replace=T )

to generate the random numbers, where 8890, 9500 and 1136 are the
minimum  and maximum of the signal and 1136 the length of sample data.
Q1: Please how do I generate many samples as x above, say up to 5000
or 10,000? I manually generated and stored as x1,x2, x3 up to x100.

Q2: Please how do I use this randomly generated numbers to test the
statistical significance level of the signal generated by
plot(-5:10,oomean,type="l",ylim=c(8890,9100),  )?

I wish to test for 90% and 99% percentile.

I am sorry that this is too long.

Many thanks for your kind contributions

Best
Ogbos







On Sun, Feb 10, 2019 at 3:55 PM Ogbos Okike <[hidden email]> wrote:

>
> Dear Michael,
> This is great! Thank you.
>
> I have not really got any response other than yours.
>
> I have long before now included what I have in a paper submitted to a journal.
>
> I am awaiting the feedback of the reviewer. I will compare the
> comments with your input here and determine the corrections to make
> and probably return to the list for additional help.
>
> Best wishes
> Ogbos
>
> On Fri, Feb 8, 2019 at 4:31 PM Meyners, Michael <[hidden email]> wrote:
> >
> > Ogbos,
> >
> > You do not seem to have received a reply over the list yet, which might be due to the fact that this seems rather a stats than an R question. Neither got your attachment (Figure) through - see posting guide.
> >
> > I'm not familiar with epoch analysis, so not sure what exactly you are doing / trying to achieve, but some general thoughts:
> >
> > * You do NOT want to restrict your re-randomizations in a way that "none of the dates corresponds with the ones in the real event" - actually, as a general principle, the true data must be an admissible re-randomization as well. You seem to have excluded that (and a lot of other randomizations at the same time which might have occurred, i.e. dates 1 and 2 reversed but all others the same), thereby rendering the test invalid. Any restrictions you have on your re-randomizations must've applied to the original randomization as well.
> > * If you have rather observational data (which I suspect, but not sure), Edgington & Onghena (2007) would rather refer to this as a permutation test - the difference being that you have to make strong assumptions (similar to parametric tests) on the nature of the data, which are designed-in to be true for randomization tests. It might be a merely linguistic discrimination, but it is important to note which assumptions have to be (implicitly) made.
> > * I'm not sure what you mean by "mean differences" of the events - is that two groups you are comparing? If so, that seems reasonable, but just make sure the test statistic you use is reasonable and sensitive against the alternatives you are mostly interested in. The randomization/permutation test will never proof that, e.g., means are significantly different, but only that there is SOME difference. By selecting the appropriate test statistic, you can influence what will pop up more easily and what not, but you can never be sure (unless you make strong assumptions about everything else, like in many parametric tests).
> > * For any test statistic, you would then determine the proportion of its values among the 5000 samples where it is as large or larger than the one observed (or as small or smaller, or either, depending on the nature of the test statistic and whether you aim for a one- or a two-sided test). That is your p value. If small enough, conclude significance. At least conceptually important: The observed test statistic is always part of the re-randomization (i.e. your 5000) - so you truly only do 4999 plus the one you observed. Otherwise the test may be more or less liberal. Your p value is hence no smaller than 1/n, where n is the total number of samples you looked at (including the observed one), a p value of 0 is not possible in randomization tests (nor in other tests, of course).
> >
> > I hope this is helpful, but you will need to go through these and refer to your own setup to check whether you adhered to the principles or not, which is impossible for me to judge based on the information provided (and I won't be able to look at excessive code to check either).
> >
> > Michael
> >
> > > -----Original Message-----
> > > From: R-help <[hidden email]> On Behalf Of Ogbos Okike
> > > Sent: Montag, 28. Januar 2019 19:42
> > > To: r-help <[hidden email]>
> > > Subject: [R] Randomization Test
> > >
> > > Dear Contributors,
> > >
> > > I conducting epoch analysis. I tried to test the significance of my result using
> > > randomization test.
> > >
> > > Since I have 71 events, I randomly selected another 71 events, making sure
> > > that none of the dates in the random events corresponds with the ones in
> > > the real event.
> > >
> > > Following the code I found here
> > > (https://www.uvm.edu/~dhowell/StatPages/R/RandomizationTestsWithR/R
> > > andom2Sample/TwoIndependentSamplesR.html),
> > > I combined these two data set and used them to generate another 5000
> > > events. I then plotted the graph of the mean differences for the 5000
> > > randomly generated events. On the graph, I indicated the region of the
> > > mean difference between the real 71 epoch and the randomly selected 71
> > > epoch.
> > >
> > > Since the two tail test shows that the mean difference falls at the extreme of
> > > the randomly selected events, I concluded that my result is statistically
> > > significant.
> > >
> > >
> > >
> > > I am attaching the graph to assistance you in you suggestions.
> > >
> > > I can attach both my code and the real and randomly generated events if you
> > > ask for it.
> > >
> > > My request is that you help me to understand if I am on the right track or no.
> > > This is the first time I am doing this and except the experts decide, I am not
> > > quite sure whether I am right or not.
> > >
> > > Many thanks for your kind concern.
> > >
> > > Best
> > > Ogbos
> > > ______________________________________________
> > > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > PLEASE do read the posting guide http://www.R-project.org/posting-
> > > guide.html
> > > and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Randomization Test

Ben Tupper-2
Hi,

I'm not very clear on what you are trying to achieve, but I think you could try the following for your Q1...

> Q1: Please how do I generate many samples as x above, say up to 5000
> or 10,000? I manually generated and stored as x1,x2, x3 up to x100.

ndta = nrow(dta)
x0 = 8890
x1 = 9500
xx = seq(from = x0, to = x1, by = 1)
N_many = 50 # make 5000 etc as required
m <- sapply(   seq_len(N_many), function(i) sample(xx, ndta, replace = TRUE))

str(m)
# int [1:1136, 1:50] 9147 8904 9062 8946 9330 9056 9239 9284 9290 9441 ...

summary(as.vector(m))
#   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
#   8890    9043    9195    9196    9348    9500

m[1:5, 1:5]
#     [,1] [,2] [,3] [,4] [,5]
#[1,] 9147 9124 9341 8999 9268
#[2,] 8904 9246 9087 9041 8943
#[3,] 9062 9184 9061 9119 9350
#[4,] 8946 9242 8932 9306 9270
#[5,] 9330 8979 9437 9030 9333

Each sample set of length ndta (in this case ndta = 1136) is found in a column of the matrix.   Is that what you are looking for?

Ben

> On Feb 27, 2019, at 4:53 PM, Ogbos Okike <[hidden email]> wrote:
>
> Dear Kind List,
>
> I am still battling with this. I have, however, made some progress
> with the suggestions of Micheal and others. At least, I have a better
> picture of what I want to do now as I will attempt a detailed
> description here.
>
> I am aware I should show you just a small part of my code and data.
> But when I copied out a small portion and run to see what you get when
> I send that,  I was not satisfied with the signal displayed. The epoch
> analysis averages data and is quite sensitive to leveraging,
> especially if a small sample is used.
>
> So please permit/exercise patience  me to display the series of epoch
> that give the averaged valued used. You can just run the code and see
> the signal of interest. Here is the code and the data:
>
> dta <- read.table( text ="n CR
> -5 8969
> -4 8932
> -3 8929
> -2 8916
> -1 8807
> 0 8449
> 1 8484
> 2 8148
> 3 8282
> 4 8305
> 5 8380
> 6 8530
> 7 8642
> 8 8780
> 9 8890
> 10 8962
> -5 8929
> -4 8916
> -3 8807
> -2 8449
> -1 8484
> 0 8148
> 1 8282
> 2 8305
> 3 8380
> 4 8530
> 5 8642
> 6 8780
> 7 8890
> 8 8962
> 9 8949
> 10 8974
> -5 8744
> -4 8786
> -3 8828
> -2 8807
> -1 8716
> 0 8520
> 1 8634
> 2 8640
> 3 8636
> 4 8658
> 5 8699
> 6 8682
> 7 8621
> 8 8626
> 9 8660
> 10 8737
> -5 8592
> -4 8612
> -3 8628
> -2 8589
> -1 8318
> 0 8264
> 1 8294
> 2 8410
> 3 8442
> 4 8416
> 5 8389
> 6 8412
> 7 8453
> 8 8563
> 9 8581
> 10 8613
> -5 8264
> -4 8294
> -3 8410
> -2 8442
> -1 8416
> 0 8389
> 1 8412
> 2 8453
> 3 8563
> 4 8581
> 5 8613
> 6 8647
> 7 8613
> 8 8508
> 9 7829
> 10 7499
> -5 8613
> -4 8647
> -3 8613
> -2 8508
> -1 7829
> 0 7499
> 1 8213
> 2 7993
> 3 7821
> 4 8316
> 5 8460
> 6 8533
> 7 8584
> 8 8586
> 9 8567
> 10 8573
> -5 8508
> -4 7829
> -3 7499
> -2 8213
> -1 7993
> 0 7821
> 1 8316
> 2 8460
> 3 8533
> 4 8584
> 5 8586
> 6 8567
> 7 8573
> 8 8617
> 9 8591
> 10 8661
> -5 8851
> -4 8893
> -3 8858
> -2 8803
> -1 8790
> 0 8468
> 1 8545
> 2 8570
> 3 8568
> 4 8624
> 5 8669
> 6 8236
> 7 8190
> 8 8313
> 9 8389
> 10 8421
> -5 8803
> -4 8790
> -3 8468
> -2 8545
> -1 8570
> 0 8568
> 1 8624
> 2 8669
> 3 8236
> 4 8190
> 5 8313
> 6 8389
> 7 8421
> 8 8468
> 9 8537
> 10 8580
> -5 8570
> -4 8568
> -3 8624
> -2 8669
> -1 8236
> 0 8190
> 1 8313
> 2 8389
> 3 8421
> 4 8468
> 5 8537
> 6 8580
> 7 8605
> 8 8646
> 9 8690
> 10 8770
> -5 8690
> -4 8770
> -3 8799
> -2 8821
> -1 8666
> 0 8539
> 1 8633
> 2 8617
> 3 8651
> 4 8693
> 5 8715
> 6 8738
> 7 8716
> 8 8677
> 9 8680
> 10 8700
> -5 8756
> -4 8632
> -3 8662
> -2 8596
> -1 8552
> 0 8502
> 1 8633
> 2 8702
> 3 8745
> 4 8730
> 5 8708
> 6 8817
> 7 8724
> 8 8688
> 9 8693
> 10 8746
> -5 8926
> -4 8888
> -3 8798
> -2 8651
> -1 8678
> 0 8578
> 1 8593
> 2 8598
> 3 8526
> 4 8181
> 5 8204
> 6 8373
> 7 8599
> 8 8773
> 9 8784
> 10 8746
> -5 8678
> -4 8578
> -3 8593
> -2 8598
> -1 8526
> 0 8181
> 1 8204
> 2 8373
> 3 8599
> 4 8773
> 5 8784
> 6 8746
> 7 8747
> 8 8757
> 9 8749
> 10 8767
> -5 8757
> -4 8749
> -3 8767
> -2 8754
> -1 8695
> 0 8631
> 1 8661
> 2 8653
> 3 8588
> 4 8562
> 5 8613
> 6 8595
> 7 8498
> 8 8404
> 9 8507
> 10 8599
> -5 8695
> -4 8631
> -3 8661
> -2 8653
> -1 8588
> 0 8562
> 1 8613
> 2 8595
> 3 8498
> 4 8404
> 5 8507
> 6 8599
> 7 8592
> 8 8600
> 9 8637
> 10 8635
> -5 8588
> -4 8562
> -3 8613
> -2 8595
> -1 8498
> 0 8404
> 1 8507
> 2 8599
> 3 8592
> 4 8600
> 5 8637
> 6 8635
> 7 8632
> 8 8674
> 9 8644
> 10 8687
> -5 8595
> -4 8498
> -3 8404
> -2 8507
> -1 8599
> 0 8592
> 1 8600
> 2 8637
> 3 8635
> 4 8632
> 5 8674
> 6 8644
> 7 8687
> 8 8721
> 9 8747
> 10 8748
> -5 8599
> -4 8592
> -3 8600
> -2 8637
> -1 8635
> 0 8632
> 1 8674
> 2 8644
> 3 8687
> 4 8721
> 5 8747
> 6 8748
> 7 8739
> 8 8763
> 9 8792
> 10 8558
> -5 8600
> -4 8637
> -3 8635
> -2 8632
> -1 8674
> 0 8644
> 1 8687
> 2 8721
> 3 8747
> 4 8748
> 5 8739
> 6 8763
> 7 8792
> 8 8558
> 9 8442
> 10 8555
> -5 8748
> -4 8739
> -3 8763
> -2 8792
> -1 8558
> 0 8442
> 1 8555
> 2 8622
> 3 8634
> 4 8698
> 5 8732
> 6 8713
> 7 8732
> 8 8681
> 9 8615
> 10 8624
> -5 8698
> -4 8732
> -3 8713
> -2 8732
> -1 8681
> 0 8615
> 1 8624
> 2 8649
> 3 8656
> 4 8678
> 5 8723
> 6 8693
> 7 8548
> 8 7803
> 9 7801
> 10 7724
> -5 8723
> -4 8693
> -3 8548
> -2 7803
> -1 7801
> 0 7724
> 1 7910
> 2 7829
> 3 7995
> 4 8156
> 5 8307
> 6 8377
> 7 8465
> 8 8506
> 9 8516
> 10 8536
> -5 8548
> -4 7803
> -3 7801
> -2 7724
> -1 7910
> 0 7829
> 1 7995
> 2 8156
> 3 8307
> 4 8377
> 5 8465
> 6 8506
> 7 8516
> 8 8536
> 9 8574
> 10 8623
> -5 8821
> -4 8856
> -3 8798
> -2 8772
> -1 8705
> 0 8682
> 1 8691
> 2 8720
> 3 8727
> 4 8789
> 5 8821
> 6 8811
> 7 8841
> 8 8849
> 9 8849
> 10 8860
> -5 8835
> -4 8829
> -3 8826
> -2 8799
> -1 8775
> 0 8756
> 1 8793
> 2 8814
> 3 8847
> 4 8838
> 5 8833
> 6 8841
> 7 8847
> 8 8903
> 9 8933
> 10 8918
> -5 8890
> -4 8875
> -3 8874
> -2 8865
> -1 8891
> 0 8839
> 1 8853
> 2 8888
> 3 8884
> 4 8890
> 5 8889
> 6 8839
> 7 8879
> 8 8908
> 9 8924
> 10 8882
> -5 8853
> -4 8888
> -3 8884
> -2 8890
> -1 8889
> 0 8839
> 1 8879
> 2 8908
> 3 8924
> 4 8882
> 5 8910
> 6 8903
> 7 8859
> 8 8858
> 9 8863
> 10 8847
> -5 8924
> -4 8882
> -3 8910
> -2 8903
> -1 8859
> 0 8858
> 1 8863
> 2 8847
> 3 8883
> 4 8869
> 5 8878
> 6 8897
> 7 8922
> 8 8895
> 9 8858
> 10 8858
> -5 8910
> -4 8903
> -3 8859
> -2 8858
> -1 8863
> 0 8847
> 1 8883
> 2 8869
> 3 8878
> 4 8897
> 5 8922
> 6 8895
> 7 8858
> 8 8858
> 9 8736
> 10 8905
> -5 8859
> -4 8858
> -3 8863
> -2 8847
> -1 8883
> 0 8869
> 1 8878
> 2 8897
> 3 8922
> 4 8895
> 5 8858
> 6 8858
> 7 8736
> 8 8905
> 9 8935
> 10 8974
> -5 8897
> -4 8922
> -3 8895
> -2 8858
> -1 8858
> 0 8736
> 1 8905
> 2 8935
> 3 8974
> 4 8946
> 5 8952
> 6 9010
> 7 8980
> 8 8976
> 9 8970
> 10 8961
> -5 9376
> -4 9336
> -3 9311
> -2 9287
> -1 9221
> 0 9087
> 1 9132
> 2 9175
> 3 9166
> 4 9240
> 5 9264
> 6 9271
> 7 9319
> 8 9324
> 9 9333
> 10 9351
> -5 9287
> -4 9221
> -3 9087
> -2 9132
> -1 9175
> 0 9166
> 1 9240
> 2 9264
> 3 9271
> 4 9319
> 5 9324
> 6 9333
> 7 9351
> 8 9362
> 9 9385
> 10 9354
> -5 9407
> -4 9414
> -3 9354
> -2 9298
> -1 9319
> 0 9147
> 1 9178
> 2 9196
> 3 9258
> 4 9303
> 5 9369
> 6 9382
> 7 9375
> 8 9389
> 9 9376
> 10 9264
> -5 9386
> -4 9396
> -3 9424
> -2 9391
> -1 9284
> 0 9267
> 1 9278
> 2 9318
> 3 9334
> 4 9275
> 5 9306
> 6 9308
> 7 9358
> 8 9335
> 9 9373
> 10 9379
> -5 9284
> -4 9267
> -3 9278
> -2 9318
> -1 9334
> 0 9275
> 1 9306
> 2 9308
> 3 9358
> 4 9335
> 5 9373
> 6 9379
> 7 9355
> 8 9340
> 9 9327
> 10 9320
> -5 9327
> -4 9320
> -3 9315
> -2 9336
> -1 9371
> 0 9259
> 1 9330
> 2 9355
> 3 9334
> 4 9353
> 5 9370
> 6 9394
> 7 9400
> 8 9318
> 9 9037
> 10 8994
> -5 9394
> -4 9400
> -3 9318
> -2 9037
> -1 8994
> 0 8943
> 1 8964
> 2 8997
> 3 9158
> 4 8964
> 5 8564
> 6 8736
> 7 8818
> 8 8938
> 9 9034
> 10 9132
> -5 8943
> -4 8964
> -3 8997
> -2 9158
> -1 8964
> 0 8564
> 1 8736
> 2 8818
> 3 8938
> 4 9034
> 5 9132
> 6 9167
> 7 9200
> 8 9257
> 9 9266
> 10 9306
> -5 9338
> -4 9354
> -3 9372
> -2 9338
> -1 9308
> 0 9282
> 1 9324
> 2 9318
> 3 9342
> 4 9370
> 5 9331
> 6 9327
> 7 9338
> 8 9381
> 9 9394
> 10 9332
> -5 9372
> -4 9338
> -3 9308
> -2 9282
> -1 9324
> 0 9318
> 1 9342
> 2 9370
> 3 9331
> 4 9327
> 5 9338
> 6 9381
> 7 9394
> 8 9332
> 9 9331
> 10 9293
> -5 9338
> -4 9381
> -3 9394
> -2 9332
> -1 9331
> 0 9293
> 1 9309
> 2 9325
> 3 9406
> 4 9409
> 5 9413
> 6 9426
> 7 9440
> 8 9449
> 9 9512
> 10 9494
> -5 9361
> -4 9354
> -3 9299
> -2 9282
> -1 9250
> 0 9242
> 1 9254
> 2 9321
> 3 9390
> 4 9414
> 5 9435
> 6 9437
> 7 9426
> 8 9398
> 9 9383
> 10 9354
> -5 9365
> -4 9421
> -3 9416
> -2 9355
> -1 9338
> 0 9324
> 1 9325
> 2 9322
> 3 9319
> 4 9381
> 5 9315
> 6 9314
> 7 9359
> 8 9403
> 9 9419
> 10 9474
> -5 9355
> -4 9338
> -3 9324
> -2 9325
> -1 9322
> 0 9319
> 1 9381
> 2 9315
> 3 9314
> 4 9359
> 5 9403
> 6 9419
> 7 9474
> 8 9525
> 9 9501
> 10 9447
> -5 9325
> -4 9322
> -3 9319
> -2 9381
> -1 9315
> 0 9314
> 1 9359
> 2 9403
> 3 9419
> 4 9474
> 5 9525
> 6 9501
> 7 9447
> 8 9424
> 9 9396
> 10 9388
> -5 9447
> -4 9424
> -3 9396
> -2 9388
> -1 9396
> 0 9346
> 1 9358
> 2 9353
> 3 9350
> 4 9378
> 5 9372
> 6 9354
> 7 9349
> 8 9392
> 9 9440
> 10 9467
> -5 9388
> -4 9396
> -3 9346
> -2 9358
> -1 9353
> 0 9350
> 1 9378
> 2 9372
> 3 9354
> 4 9349
> 5 9392
> 6 9440
> 7 9467
> 8 9519
> 9 9550
> 10 9565
> -5 9353
> -4 9350
> -3 9378
> -2 9372
> -1 9354
> 0 9349
> 1 9392
> 2 9440
> 3 9467
> 4 9519
> 5 9550
> 6 9565
> 7 9565
> 8 9497
> 9 9500
> 10 9472
> -5 9522
> -4 9529
> -3 9492
> -2 9432
> -1 9382
> 0 9355
> 1 9361
> 2 9350
> 3 9382
> 4 9451
> 5 9491
> 6 9506
> 7 9529
> 8 9543
> 9 9556
> 10 9553
> -5 9492
> -4 9432
> -3 9382
> -2 9355
> -1 9361
> 0 9350
> 1 9382
> 2 9451
> 3 9491
> 4 9506
> 5 9529
> 6 9543
> 7 9556
> 8 9553
> 9 9502
> 10 9470
> -5 9551
> -4 9505
> -3 9389
> -2 9406
> -1 9377
> 0 9284
> 1 9365
> 2 9424
> 3 9412
> 4 9403
> 5 9384
> 6 9394
> 7 9404
> 8 9413
> 9 9407
> 10 9405
> -5 9579
> -4 9576
> -3 9543
> -2 9451
> -1 9421
> 0 9361
> 1 9394
> 2 9400
> 3 9387
> 4 9366
> 5 9346
> 6 9360
> 7 9385
> 8 9435
> 9 9443
> 10 9430
> -5 9361
> -4 9394
> -3 9400
> -2 9387
> -1 9366
> 0 9346
> 1 9360
> 2 9385
> 3 9435
> 4 9443
> 5 9430
> 6 9454
> 7 9531
> 8 9547
> 9 9581
> 10 9540
> -5 9510
> -4 9546
> -3 9564
> -2 9508
> -1 9422
> 0 9369
> 1 9395
> 2 9438
> 3 9423
> 4 9392
> 5 9368
> 6 9366
> 7 9348
> 8 9340
> 9 9375
> 10 9391
> -5 9423
> -4 9392
> -3 9368
> -2 9366
> -1 9348
> 0 9340
> 1 9375
> 2 9391
> 3 9466
> 4 9545
> 5 9574
> 6 9564
> 7 9527
> 8 9513
> 9 9494
> 10 9542
> -5 9511
> -4 9491
> -3 9457
> -2 9453
> -1 9402
> 0 9382
> 1 9407
> 2 9437
> 3 9403
> 4 9404
> 5 9425
> 6 9486
> 7 9457
> 8 9451
> 9 9423
> 10 9401
> -5 9425
> -4 9486
> -3 9457
> -2 9451
> -1 9423
> 0 9401
> 1 9429
> 2 9422
> 3 9431
> 4 9462
> 5 9475
> 6 9474
> 7 9487
> 8 9493
> 9 9495
> 10 9499
> -5 9404
> -4 9385
> -3 9363
> -2 9399
> -1 9411
> 0 9355
> 1 9357
> 2 9363
> 3 9382
> 4 9387
> 5 9408
> 6 9429
> 7 9456
> 8 9487
> 9 9526
> 10 9487
> -5 9493
> -4 9439
> -3 9400
> -2 9378
> -1 9371
> 0 9369
> 1 9374
> 2 9305
> 3 9298
> 4 9298
> 5 9325
> 6 9381
> 7 9477
> 8 9508
> 9 9496
> 10 9517
> -5 9371
> -4 9369
> -3 9374
> -2 9305
> -1 9298
> 0 9298
> 1 9325
> 2 9381
> 3 9477
> 4 9508
> 5 9496
> 6 9517
> 7 9561
> 8 9570
> 9 9546
> 10 9544
> -5 9510
> -4 9506
> -3 9530
> -2 9441
> -1 9427
> 0 9393
> 1 9420
> 2 9444
> 3 9468
> 4 9484
> 5 9525
> 6 9542
> 7 9557
> 8 9548
> 9 9550
> 10 9593
> -5 9589
> -4 9598
> -3 9527
> -2 9417
> -1 9390
> 0 9374
> 1 9386
> 2 9407
> 3 9453
> 4 9447
> 5 9419
> 6 9386
> 7 9373
> 8 9364
> 9 9376
> 10 9389
> -5 9453
> -4 9447
> -3 9419
> -2 9386
> -1 9373
> 0 9364
> 1 9376
> 2 9389
> 3 9376
> 4 9375
> 5 9370
> 6 9391
> 7 9458
> 8 9446
> 9 9456
> 10 9463
> -5 9364
> -4 9376
> -3 9389
> -2 9376
> -1 9375
> 0 9370
> 1 9391
> 2 9458
> 3 9446
> 4 9456
> 5 9463
> 6 9500
> 7 9486
> 8 9474
> 9 9495
> 10 9531
> -5 9491
> -4 9441
> -3 9388
> -2 9380
> -1 9369
> 0 9354
> 1 9367
> 2 9369
> 3 9341
> 4 9305
> 5 9308
> 6 9324
> 7 9385
> 8 9451
> 9 9496
> 10 9527
> -5 9369
> -4 9354
> -3 9367
> -2 9369
> -1 9341
> 0 9305
> 1 9308
> 2 9324
> 3 9385
> 4 9451
> 5 9496
> 6 9527
> 7 9544
> 8 9543
> 9 9535
> 10 9536
> -5 9586
> -4 9583
> -3 9572
> -2 9533
> -1 9454
> 0 9392
> 1 9420
> 2 9451
> 3 9475
> 4 9514
> 5 9561
> 6 9542
> 7 9502
> 8 9461
> 9 9468
> 10 9463
> -5 9587
> -4 9562
> -3 9530
> -2 9445
> -1 9404
> 0 9395
> 1 9417
> 2 9449
> 3 9467
> 4 9470
> 5 9524
> 6 9512
> 7 9448
> 8 9398
> 9 9431
> 10 9467
> -5 9467
> -4 9470
> -3 9524
> -2 9512
> -1 9448
> 0 9398
> 1 9431
> 2 9467
> 3 9490
> 4 9517
> 5 9526
> 6 9574
> 7 9573
> 8 9562
> 9 9563
> 10 9566
> ",header=TRUE)
>
> data<-matrix(c(dta$CR),ncol=71)
> A<-matrix(rep(-5:10,71))
> B<-matrix(data)
>
> oodf<-data.frame(A,B)
> a<--5:10
> oodf<-data.frame(A,B)
> library(plotrix)
> std.error<-function(x) return(sd(x)/(sum(!is.na(x))))
> oomean<-as.vector(by(oodf$B,oodf$A,mean))
> oose<-as.vector(by(oodf$B,oodf$A,std.error))
> plot(-5:10,oomean,type="l",ylim=c(8890,9100),
> )
> A<-oomean-1.96*oose
> B<-oomean+1.96*oose
> lines(a,A,col="red")
> lines(a,B,col="red")
>
> My Question:
> I wish to conduct a randomization test of significance (90 and 99
> percentile) of the reductions/decreases as displayed by the signal.
>
> I am attempting using:
> x<-sample(8890:9500,1136,replace=T )
>
> to generate the random numbers, where 8890, 9500 and 1136 are the
> minimum  and maximum of the signal and 1136 the length of sample data.
> Q1: Please how do I generate many samples as x above, say up to 5000
> or 10,000? I manually generated and stored as x1,x2, x3 up to x100.
>
> Q2: Please how do I use this randomly generated numbers to test the
> statistical significance level of the signal generated by
> plot(-5:10,oomean,type="l",ylim=c(8890,9100),  )?
>
> I wish to test for 90% and 99% percentile.
>
> I am sorry that this is too long.
>
> Many thanks for your kind contributions
>
> Best
> Ogbos
>
>
>
>
>
>
>
> On Sun, Feb 10, 2019 at 3:55 PM Ogbos Okike <[hidden email]> wrote:
>>
>> Dear Michael,
>> This is great! Thank you.
>>
>> I have not really got any response other than yours.
>>
>> I have long before now included what I have in a paper submitted to a journal.
>>
>> I am awaiting the feedback of the reviewer. I will compare the
>> comments with your input here and determine the corrections to make
>> and probably return to the list for additional help.
>>
>> Best wishes
>> Ogbos
>>
>> On Fri, Feb 8, 2019 at 4:31 PM Meyners, Michael <[hidden email]> wrote:
>>>
>>> Ogbos,
>>>
>>> You do not seem to have received a reply over the list yet, which might be due to the fact that this seems rather a stats than an R question. Neither got your attachment (Figure) through - see posting guide.
>>>
>>> I'm not familiar with epoch analysis, so not sure what exactly you are doing / trying to achieve, but some general thoughts:
>>>
>>> * You do NOT want to restrict your re-randomizations in a way that "none of the dates corresponds with the ones in the real event" - actually, as a general principle, the true data must be an admissible re-randomization as well. You seem to have excluded that (and a lot of other randomizations at the same time which might have occurred, i.e. dates 1 and 2 reversed but all others the same), thereby rendering the test invalid. Any restrictions you have on your re-randomizations must've applied to the original randomization as well.
>>> * If you have rather observational data (which I suspect, but not sure), Edgington & Onghena (2007) would rather refer to this as a permutation test - the difference being that you have to make strong assumptions (similar to parametric tests) on the nature of the data, which are designed-in to be true for randomization tests. It might be a merely linguistic discrimination, but it is important to note which assumptions have to be (implicitly) made.
>>> * I'm not sure what you mean by "mean differences" of the events - is that two groups you are comparing? If so, that seems reasonable, but just make sure the test statistic you use is reasonable and sensitive against the alternatives you are mostly interested in. The randomization/permutation test will never proof that, e.g., means are significantly different, but only that there is SOME difference. By selecting the appropriate test statistic, you can influence what will pop up more easily and what not, but you can never be sure (unless you make strong assumptions about everything else, like in many parametric tests).
>>> * For any test statistic, you would then determine the proportion of its values among the 5000 samples where it is as large or larger than the one observed (or as small or smaller, or either, depending on the nature of the test statistic and whether you aim for a one- or a two-sided test). That is your p value. If small enough, conclude significance. At least conceptually important: The observed test statistic is always part of the re-randomization (i.e. your 5000) - so you truly only do 4999 plus the one you observed. Otherwise the test may be more or less liberal. Your p value is hence no smaller than 1/n, where n is the total number of samples you looked at (including the observed one), a p value of 0 is not possible in randomization tests (nor in other tests, of course).
>>>
>>> I hope this is helpful, but you will need to go through these and refer to your own setup to check whether you adhered to the principles or not, which is impossible for me to judge based on the information provided (and I won't be able to look at excessive code to check either).
>>>
>>> Michael
>>>
>>>> -----Original Message-----
>>>> From: R-help <[hidden email]> On Behalf Of Ogbos Okike
>>>> Sent: Montag, 28. Januar 2019 19:42
>>>> To: r-help <[hidden email]>
>>>> Subject: [R] Randomization Test
>>>>
>>>> Dear Contributors,
>>>>
>>>> I conducting epoch analysis. I tried to test the significance of my result using
>>>> randomization test.
>>>>
>>>> Since I have 71 events, I randomly selected another 71 events, making sure
>>>> that none of the dates in the random events corresponds with the ones in
>>>> the real event.
>>>>
>>>> Following the code I found here
>>>> (https://www.uvm.edu/~dhowell/StatPages/R/RandomizationTestsWithR/R
>>>> andom2Sample/TwoIndependentSamplesR.html),
>>>> I combined these two data set and used them to generate another 5000
>>>> events. I then plotted the graph of the mean differences for the 5000
>>>> randomly generated events. On the graph, I indicated the region of the
>>>> mean difference between the real 71 epoch and the randomly selected 71
>>>> epoch.
>>>>
>>>> Since the two tail test shows that the mean difference falls at the extreme of
>>>> the randomly selected events, I concluded that my result is statistically
>>>> significant.
>>>>
>>>>
>>>>
>>>> I am attaching the graph to assistance you in you suggestions.
>>>>
>>>> I can attach both my code and the real and randomly generated events if you
>>>> ask for it.
>>>>
>>>> My request is that you help me to understand if I am on the right track or no.
>>>> This is the first time I am doing this and except the experts decide, I am not
>>>> quite sure whether I am right or not.
>>>>
>>>> Many thanks for your kind concern.
>>>>
>>>> Best
>>>> Ogbos
>>>> ______________________________________________
>>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>> PLEASE do read the posting guide http://www.R-project.org/posting-
>>>> guide.html
>>>> and provide commented, minimal, self-contained, reproducible code.
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

Ben Tupper
Bigelow Laboratory for Ocean Sciences
60 Bigelow Drive, P.O. Box 380
East Boothbay, Maine 04544
http://www.bigelow.org

Ecological Forecasting: https://eco.bigelow.org/

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Randomization Test

Meyners, Michael
In reply to this post by Ogbos
Ogbos,

To share data (in particularly lengthy one as yours), check out ?dput

To replicate sampling, look at ?replicate (will output to a data frame to use further) - that should answer your Q1.

Apart from that (and regarding Q2), the task you are after is getting more and more obscure to me. I don't see what you really want to test - between levels of n (or A in oodf)? If so, you would need to permute levels within each set (are they sets of -5:10, or are all observations completely independent? In the latter case, across all sets). What is the null hypothesis you want to test? And what is the data exactly? I don't understand what the two columns indicate. Then you need to decide on what the test statistic is you want to use. The average? The difference between each pair of averages? Anything like that? Do you want to test all levels simultaneously, or by pairs?

Clearly, your way of sampling is inappropriate for a randomization test. You are rather simulating data that can take any value between min and max with equal probability. That does not seem to be the right null model to me (it might be, though, depending on your null hypothesis). It would not make a randomization test either way, but rather a Monte Carlo simulation under the null hypothesis. Then you would determine your test statistic(s) (whichever they are) and subsequently repeat that often enough and check whether your observed value is higher than 90 or 99% of the simulated ones. Again, that would be a simulation, not a randomization test.

For the latter, you'd need to permute the data in an appropriate way (again depending on the null hypothesis, and on the structure, i.e. are they are coming in blocks of any kind, or all independent), and then recalculate the test statistic every time, and then proceed as before. I'm not sure whether the data was the result of a randomized experiment - if not, you can still use the idea, but fall into something that some refer to as "permutation testing" - the difference being that you need to make much stronger assumptions on independence etc of the data. Many use the terms equivalently, but just so you are aware.

I really think you need to look into a textbook (eg. Edgington & Onghena) or some papers to understand the concept of randomization tests, or consult with a statistician with good background in that field. What you are suggesting is not near it, and unless you have a clear hypothesis and a good understanding of how the data was generated, it is impossible for you (and anyone else) to say how such a test might be designed.

Michael


> -----Original Message-----
> From: Ogbos Okike <[hidden email]>
> Sent: Mittwoch, 27. Februar 2019 22:53
> To: Meyners, Michael <[hidden email]>
> Cc: r-help <[hidden email]>
> Subject: Re: [R] Randomization Test
>
> Dear Kind List,
>
> I am still battling with this. I have, however, made some progress with the
> suggestions of Micheal and others. At least, I have a better picture of what I
> want to do now as I will attempt a detailed description here.
>
> I am aware I should show you just a small part of my code and data.
> But when I copied out a small portion and run to see what you get when I
> send that,  I was not satisfied with the signal displayed. The epoch analysis
> averages data and is quite sensitive to leveraging, especially if a small sample
> is used.
>
> So please permit/exercise patience  me to display the series of epoch that
> give the averaged valued used. You can just run the code and see the signal
> of interest. Here is the code and the data:
>
> dta <- read.table( text ="n CR
>  -5 8969
.
SNIP...
.

> 10 9566
> ",header=TRUE)
>
>  data<-matrix(c(dta$CR),ncol=71)
> A<-matrix(rep(-5:10,71))
> B<-matrix(data)
>
>  oodf<-data.frame(A,B)
>  a<--5:10
> oodf<-data.frame(A,B)
> library(plotrix)
> std.error<-function(x) return(sd(x)/(sum(!is.na(x))))
> oomean<-as.vector(by(oodf$B,oodf$A,mean))
> oose<-as.vector(by(oodf$B,oodf$A,std.error))
> plot(-5:10,oomean,type="l",ylim=c(8890,9100),
>  )
> A<-oomean-1.96*oose
>  B<-oomean+1.96*oose
> lines(a,A,col="red")
>  lines(a,B,col="red")
>
>  My Question:
> I wish to conduct a randomization test of significance (90 and 99
> percentile) of the reductions/decreases as displayed by the signal.
>
> I am attempting using:
> x<-sample(8890:9500,1136,replace=T )
>
> to generate the random numbers, where 8890, 9500 and 1136 are the
> minimum  and maximum of the signal and 1136 the length of sample data.
> Q1: Please how do I generate many samples as x above, say up to 5000 or
> 10,000? I manually generated and stored as x1,x2, x3 up to x100.
>
> Q2: Please how do I use this randomly generated numbers to test the
> statistical significance level of the signal generated by plot(-
> 5:10,oomean,type="l",ylim=c(8890,9100),  )?
>
> I wish to test for 90% and 99% percentile.
>
> I am sorry that this is too long.
>
> Many thanks for your kind contributions
>
> Best
> Ogbos
>
>
>
>
>
>
>
> On Sun, Feb 10, 2019 at 3:55 PM Ogbos Okike <[hidden email]>
> wrote:
> >
> > Dear Michael,
> > This is great! Thank you.
> >
> > I have not really got any response other than yours.
> >
> > I have long before now included what I have in a paper submitted to a
> journal.
> >
> > I am awaiting the feedback of the reviewer. I will compare the
> > comments with your input here and determine the corrections to make
> > and probably return to the list for additional help.
> >
> > Best wishes
> > Ogbos
> >
> > On Fri, Feb 8, 2019 at 4:31 PM Meyners, Michael <[hidden email]>
> wrote:
> > >
> > > Ogbos,
> > >
> > > You do not seem to have received a reply over the list yet, which might
> be due to the fact that this seems rather a stats than an R question. Neither
> got your attachment (Figure) through - see posting guide.
> > >
> > > I'm not familiar with epoch analysis, so not sure what exactly you are
> doing / trying to achieve, but some general thoughts:
> > >
> > > * You do NOT want to restrict your re-randomizations in a way that "none
> of the dates corresponds with the ones in the real event" - actually, as a
> general principle, the true data must be an admissible re-randomization as
> well. You seem to have excluded that (and a lot of other randomizations at
> the same time which might have occurred, i.e. dates 1 and 2 reversed but all
> others the same), thereby rendering the test invalid. Any restrictions you
> have on your re-randomizations must've applied to the original
> randomization as well.
> > > * If you have rather observational data (which I suspect, but not sure),
> Edgington & Onghena (2007) would rather refer to this as a permutation test
> - the difference being that you have to make strong assumptions (similar to
> parametric tests) on the nature of the data, which are designed-in to be true
> for randomization tests. It might be a merely linguistic discrimination, but it is
> important to note which assumptions have to be (implicitly) made.
> > > * I'm not sure what you mean by "mean differences" of the events - is
> that two groups you are comparing? If so, that seems reasonable, but just
> make sure the test statistic you use is reasonable and sensitive against the
> alternatives you are mostly interested in. The randomization/permutation
> test will never proof that, e.g., means are significantly different, but only that
> there is SOME difference. By selecting the appropriate test statistic, you can
> influence what will pop up more easily and what not, but you can never be
> sure (unless you make strong assumptions about everything else, like in
> many parametric tests).
> > > * For any test statistic, you would then determine the proportion of its
> values among the 5000 samples where it is as large or larger than the one
> observed (or as small or smaller, or either, depending on the nature of the
> test statistic and whether you aim for a one- or a two-sided test). That is your
> p value. If small enough, conclude significance. At least conceptually
> important: The observed test statistic is always part of the re-randomization
> (i.e. your 5000) - so you truly only do 4999 plus the one you observed.
> Otherwise the test may be more or less liberal. Your p value is hence no
> smaller than 1/n, where n is the total number of samples you looked at
> (including the observed one), a p value of 0 is not possible in randomization
> tests (nor in other tests, of course).
> > >
> > > I hope this is helpful, but you will need to go through these and refer to
> your own setup to check whether you adhered to the principles or not,
> which is impossible for me to judge based on the information provided (and I
> won't be able to look at excessive code to check either).
> > >
> > > Michael
> > >
> > > > -----Original Message-----
> > > > From: R-help <[hidden email]> On Behalf Of Ogbos
> > > > Okike
> > > > Sent: Montag, 28. Januar 2019 19:42
> > > > To: r-help <[hidden email]>
> > > > Subject: [R] Randomization Test
> > > >
> > > > Dear Contributors,
> > > >
> > > > I conducting epoch analysis. I tried to test the significance of
> > > > my result using randomization test.
> > > >
> > > > Since I have 71 events, I randomly selected another 71 events,
> > > > making sure that none of the dates in the random events
> > > > corresponds with the ones in the real event.
> > > >
> > > > Following the code I found here
> > > >
> (https://www.uvm.edu/~dhowell/StatPages/R/RandomizationTestsWithR/
> > > > R andom2Sample/TwoIndependentSamplesR.html),
> > > > I combined these two data set and used them to generate another
> > > > 5000 events. I then plotted the graph of the mean differences for
> > > > the 5000 randomly generated events. On the graph, I indicated the
> > > > region of the mean difference between the real 71 epoch and the
> > > > randomly selected 71 epoch.
> > > >
> > > > Since the two tail test shows that the mean difference falls at
> > > > the extreme of the randomly selected events, I concluded that my
> > > > result is statistically significant.
> > > >
> > > >
> > > >
> > > > I am attaching the graph to assistance you in you suggestions.
> > > >
> > > > I can attach both my code and the real and randomly generated
> > > > events if you ask for it.
> > > >
> > > > My request is that you help me to understand if I am on the right track
> or no.
> > > > This is the first time I am doing this and except the experts
> > > > decide, I am not quite sure whether I am right or not.
> > > >
> > > > Many thanks for your kind concern.
> > > >
> > > > Best
> > > > Ogbos
> > > > ______________________________________________
> > > > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > > PLEASE do read the posting guide http://www.R-project.org/posting-
> > > > guide.html and provide commented, minimal, self-contained,
> > > > reproducible code.
______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Randomization Test: SOLVED

Ogbos
Dear Ben and Micheal,
Your contributions are quite useful to me!!!

Although my question is quite difficult to articulate, your attempt to
understand what I intend to do and my desperate efforts to get my
points across to you have, interestingly, combined to solve the
problem.

Indeed, the problem is specific (epoch/Chree analysis) and they are
yet a lot of misunderstanding even among scientists using them. I
initially felt quite odd asking you about it and it took me several
days before I decided to  post the question. Though I didn't receive
much of a response, I was happy that I was not trashed.

On the issue of textbook, my reviewer already pointed the example
he/she would like me to follow (a published article). I have but that
option if I want the paper to go soon.

So I spent about 3 days looking at the paper and similar ones. It was
in the course of trying to re-construct my question and re-post
yesterday that I got the idea of how to do it.

As Micheal pointed out, it is Monte Carlo technique that I was after.
If I am to follow the paper recommended, one hundred epoch selected
out of the original epoch will do the job. That will then be used to
judge the statistical significance of the observed decreases.

After posting the question yesterday, the problem became clearer to me
and the next few attempts got me to the destination.

I am thus indebted you.

With the very best wishes
Ogbos
On Thu, Feb 28, 2019 at 3:56 PM Meyners, Michael <[hidden email]> wrote:

>
> Ogbos,
>
> To share data (in particularly lengthy one as yours), check out ?dput
>
> To replicate sampling, look at ?replicate (will output to a data frame to use further) - that should answer your Q1.
>
> Apart from that (and regarding Q2), the task you are after is getting more and more obscure to me. I don't see what you really want to test - between levels of n (or A in oodf)? If so, you would need to permute levels within each set (are they sets of -5:10, or are all observations completely independent? In the latter case, across all sets). What is the null hypothesis you want to test? And what is the data exactly? I don't understand what the two columns indicate. Then you need to decide on what the test statistic is you want to use. The average? The difference between each pair of averages? Anything like that? Do you want to test all levels simultaneously, or by pairs?
>
> Clearly, your way of sampling is inappropriate for a randomization test. You are rather simulating data that can take any value between min and max with equal probability. That does not seem to be the right null model to me (it might be, though, depending on your null hypothesis). It would not make a randomization test either way, but rather a Monte Carlo simulation under the null hypothesis. Then you would determine your test statistic(s) (whichever they are) and subsequently repeat that often enough and check whether your observed value is higher than 90 or 99% of the simulated ones. Again, that would be a simulation, not a randomization test.
>
> For the latter, you'd need to permute the data in an appropriate way (again depending on the null hypothesis, and on the structure, i.e. are they are coming in blocks of any kind, or all independent), and then recalculate the test statistic every time, and then proceed as before. I'm not sure whether the data was the result of a randomized experiment - if not, you can still use the idea, but fall into something that some refer to as "permutation testing" - the difference being that you need to make much stronger assumptions on independence etc of the data. Many use the terms equivalently, but just so you are aware.
>
> I really think you need to look into a textbook (eg. Edgington & Onghena) or some papers to understand the concept of randomization tests, or consult with a statistician with good background in that field. What you are suggesting is not near it, and unless you have a clear hypothesis and a good understanding of how the data was generated, it is impossible for you (and anyone else) to say how such a test might be designed.
>
> Michael
>
>
> > -----Original Message-----
> > From: Ogbos Okike <[hidden email]>
> > Sent: Mittwoch, 27. Februar 2019 22:53
> > To: Meyners, Michael <[hidden email]>
> > Cc: r-help <[hidden email]>
> > Subject: Re: [R] Randomization Test
> >
> > Dear Kind List,
> >
> > I am still battling with this. I have, however, made some progress with the
> > suggestions of Micheal and others. At least, I have a better picture of what I
> > want to do now as I will attempt a detailed description here.
> >
> > I am aware I should show you just a small part of my code and data.
> > But when I copied out a small portion and run to see what you get when I
> > send that,  I was not satisfied with the signal displayed. The epoch analysis
> > averages data and is quite sensitive to leveraging, especially if a small sample
> > is used.
> >
> > So please permit/exercise patience  me to display the series of epoch that
> > give the averaged valued used. You can just run the code and see the signal
> > of interest. Here is the code and the data:
> >
> > dta <- read.table( text ="n CR
> >  -5 8969
> .
> SNIP...
> .
> > 10 9566
> > ",header=TRUE)
> >
> >  data<-matrix(c(dta$CR),ncol=71)
> > A<-matrix(rep(-5:10,71))
> > B<-matrix(data)
> >
> >  oodf<-data.frame(A,B)
> >  a<--5:10
> > oodf<-data.frame(A,B)
> > library(plotrix)
> > std.error<-function(x) return(sd(x)/(sum(!is.na(x))))
> > oomean<-as.vector(by(oodf$B,oodf$A,mean))
> > oose<-as.vector(by(oodf$B,oodf$A,std.error))
> > plot(-5:10,oomean,type="l",ylim=c(8890,9100),
> >  )
> > A<-oomean-1.96*oose
> >  B<-oomean+1.96*oose
> > lines(a,A,col="red")
> >  lines(a,B,col="red")
> >
> >  My Question:
> > I wish to conduct a randomization test of significance (90 and 99
> > percentile) of the reductions/decreases as displayed by the signal.
> >
> > I am attempting using:
> > x<-sample(8890:9500,1136,replace=T )
> >
> > to generate the random numbers, where 8890, 9500 and 1136 are the
> > minimum  and maximum of the signal and 1136 the length of sample data.
> > Q1: Please how do I generate many samples as x above, say up to 5000 or
> > 10,000? I manually generated and stored as x1,x2, x3 up to x100.
> >
> > Q2: Please how do I use this randomly generated numbers to test the
> > statistical significance level of the signal generated by plot(-
> > 5:10,oomean,type="l",ylim=c(8890,9100),  )?
> >
> > I wish to test for 90% and 99% percentile.
> >
> > I am sorry that this is too long.
> >
> > Many thanks for your kind contributions
> >
> > Best
> > Ogbos
> >
> >
> >
> >
> >
> >
> >
> > On Sun, Feb 10, 2019 at 3:55 PM Ogbos Okike <[hidden email]>
> > wrote:
> > >
> > > Dear Michael,
> > > This is great! Thank you.
> > >
> > > I have not really got any response other than yours.
> > >
> > > I have long before now included what I have in a paper submitted to a
> > journal.
> > >
> > > I am awaiting the feedback of the reviewer. I will compare the
> > > comments with your input here and determine the corrections to make
> > > and probably return to the list for additional help.
> > >
> > > Best wishes
> > > Ogbos
> > >
> > > On Fri, Feb 8, 2019 at 4:31 PM Meyners, Michael <[hidden email]>
> > wrote:
> > > >
> > > > Ogbos,
> > > >
> > > > You do not seem to have received a reply over the list yet, which might
> > be due to the fact that this seems rather a stats than an R question. Neither
> > got your attachment (Figure) through - see posting guide.
> > > >
> > > > I'm not familiar with epoch analysis, so not sure what exactly you are
> > doing / trying to achieve, but some general thoughts:
> > > >
> > > > * You do NOT want to restrict your re-randomizations in a way that "none
> > of the dates corresponds with the ones in the real event" - actually, as a
> > general principle, the true data must be an admissible re-randomization as
> > well. You seem to have excluded that (and a lot of other randomizations at
> > the same time which might have occurred, i.e. dates 1 and 2 reversed but all
> > others the same), thereby rendering the test invalid. Any restrictions you
> > have on your re-randomizations must've applied to the original
> > randomization as well.
> > > > * If you have rather observational data (which I suspect, but not sure),
> > Edgington & Onghena (2007) would rather refer to this as a permutation test
> > - the difference being that you have to make strong assumptions (similar to
> > parametric tests) on the nature of the data, which are designed-in to be true
> > for randomization tests. It might be a merely linguistic discrimination, but it is
> > important to note which assumptions have to be (implicitly) made.
> > > > * I'm not sure what you mean by "mean differences" of the events - is
> > that two groups you are comparing? If so, that seems reasonable, but just
> > make sure the test statistic you use is reasonable and sensitive against the
> > alternatives you are mostly interested in. The randomization/permutation
> > test will never proof that, e.g., means are significantly different, but only that
> > there is SOME difference. By selecting the appropriate test statistic, you can
> > influence what will pop up more easily and what not, but you can never be
> > sure (unless you make strong assumptions about everything else, like in
> > many parametric tests).
> > > > * For any test statistic, you would then determine the proportion of its
> > values among the 5000 samples where it is as large or larger than the one
> > observed (or as small or smaller, or either, depending on the nature of the
> > test statistic and whether you aim for a one- or a two-sided test). That is your
> > p value. If small enough, conclude significance. At least conceptually
> > important: The observed test statistic is always part of the re-randomization
> > (i.e. your 5000) - so you truly only do 4999 plus the one you observed.
> > Otherwise the test may be more or less liberal. Your p value is hence no
> > smaller than 1/n, where n is the total number of samples you looked at
> > (including the observed one), a p value of 0 is not possible in randomization
> > tests (nor in other tests, of course).
> > > >
> > > > I hope this is helpful, but you will need to go through these and refer to
> > your own setup to check whether you adhered to the principles or not,
> > which is impossible for me to judge based on the information provided (and I
> > won't be able to look at excessive code to check either).
> > > >
> > > > Michael
> > > >
> > > > > -----Original Message-----
> > > > > From: R-help <[hidden email]> On Behalf Of Ogbos
> > > > > Okike
> > > > > Sent: Montag, 28. Januar 2019 19:42
> > > > > To: r-help <[hidden email]>
> > > > > Subject: [R] Randomization Test
> > > > >
> > > > > Dear Contributors,
> > > > >
> > > > > I conducting epoch analysis. I tried to test the significance of
> > > > > my result using randomization test.
> > > > >
> > > > > Since I have 71 events, I randomly selected another 71 events,
> > > > > making sure that none of the dates in the random events
> > > > > corresponds with the ones in the real event.
> > > > >
> > > > > Following the code I found here
> > > > >
> > (https://www.uvm.edu/~dhowell/StatPages/R/RandomizationTestsWithR/
> > > > > R andom2Sample/TwoIndependentSamplesR.html),
> > > > > I combined these two data set and used them to generate another
> > > > > 5000 events. I then plotted the graph of the mean differences for
> > > > > the 5000 randomly generated events. On the graph, I indicated the
> > > > > region of the mean difference between the real 71 epoch and the
> > > > > randomly selected 71 epoch.
> > > > >
> > > > > Since the two tail test shows that the mean difference falls at
> > > > > the extreme of the randomly selected events, I concluded that my
> > > > > result is statistically significant.
> > > > >
> > > > >
> > > > >
> > > > > I am attaching the graph to assistance you in you suggestions.
> > > > >
> > > > > I can attach both my code and the real and randomly generated
> > > > > events if you ask for it.
> > > > >
> > > > > My request is that you help me to understand if I am on the right track
> > or no.
> > > > > This is the first time I am doing this and except the experts
> > > > > decide, I am not quite sure whether I am right or not.
> > > > >
> > > > > Many thanks for your kind concern.
> > > > >
> > > > > Best
> > > > > Ogbos
> > > > > ______________________________________________
> > > > > [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> > > > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > > > PLEASE do read the posting guide http://www.R-project.org/posting-
> > > > > guide.html and provide commented, minimal, self-contained,
> > > > > reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.