rugarch gives two different results based on the same model…how is that even possible?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

rugarch gives two different results based on the same model…how is that even possible?

GALIB KHAN
Recently I have discovered a problem with a package called rugarch that
creates arma-garch models. The issue is that if you literally change the
positions of the x variables (external regressors) then you get two
completely different results.

In other words:

   - model1 = (arma(2,2) + garch(1,0) + x1 + x2)
   - model2 = (arma(2,2) + garch(1,0) + x2 + x1)
   - rugarch's output is essentially saying that model1 != model2
   - When the correct result should be model1 == model2

I may not know a lot of statistics but I know for a fact that if you move
the x variables around, the output should still be the same.

Am I wrong on this?

Here's my stack exchange post that shows a generic R script proving my
point: Should the positioning of the external regressors change the output
of arma-garch? (Possible rugarch bug/error)
<https://stackoverflow.com/questions/51900177/should-the-positioning-of-the-external-regressors-change-the-output-of-arma-garc>

Any feedback is welcomed.

Thanks

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: rugarch gives two different results based on the same model…how is that even possible?

alexios
I run the code you provided and obtain the following results related to the external parameters:


Case 1 (x1,x2)
# x2 is second

            Estimate   Std. Error       t value  Pr(>|t|)
mxreg1  1.6724148 1.203377e-01  1.389767e+01 0.0000000
mxreg2  2.5310286 1.878833e-02  1.347128e+02 0.0000000

Case 2 (x2,x1)
# i.e. x2 is now first

mxreg1  2.5225382  0.04292725  58.7631024 0.000000e+00
mxreg2  1.6782986  0.12769622  13.1428990 0.000000e+00

Small differences in the coefficients are the result of the optimizer. There may be an issues in the
way starting parameters are being generated based on some recent input from Josh Ulrich (still to investigate)
and related to arima0 (used to generate start parameters), but otherwise don’t see a large problem at first glance.

Alexios

> On Aug 19, 2018, at 5:46 PM, GALIB KHAN <[hidden email]> wrote:
>
> Recently I have discovered a problem with a package called rugarch that
> creates arma-garch models. The issue is that if you literally change the
> positions of the x variables (external regressors) then you get two
> completely different results.
>
> In other words:
>
>   - model1 = (arma(2,2) + garch(1,0) + x1 + x2)
>   - model2 = (arma(2,2) + garch(1,0) + x2 + x1)
>   - rugarch's output is essentially saying that model1 != model2
>   - When the correct result should be model1 == model2
>
> I may not know a lot of statistics but I know for a fact that if you move
> the x variables around, the output should still be the same.
>
> Am I wrong on this?
>
> Here's my stack exchange post that shows a generic R script proving my
> point: Should the positioning of the external regressors change the output
> of arma-garch? (Possible rugarch bug/error)
> <https://stackoverflow.com/questions/51900177/should-the-positioning-of-the-external-regressors-change-the-output-of-arma-garc>
>
> Any feedback is welcomed.
>
> Thanks
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions should go.
>

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Fwd: rugarch gives two different results based on the same model…how is that even possible?

GALIB KHAN
Galib Khan
Rutgers Business School '18
Business Analytics and Information Technology
(609) 412-3654

---------- Forwarded message ----------
From: GALIB KHAN <[hidden email]>
Date: Sun, Aug 19, 2018 at 10:56 PM
Subject: Re: [R-SIG-Finance] rugarch gives two different results based on
the same model…how is that even possible?
To: alexios galanos <[hidden email]>


Alexios,

Did you set the set the seed to 1, because I'm looking at your results and
the numbers do not match with the numbers that I have provided.

I understand why the coefficients' estimates are similar but it doesn't
explain why other columns such as the t-value and pr are off by a large
margin. Also estimates for mu, ar*, ma*, omega, alpha1, and shape may have
large differences.

Take mu as an example:
-7.538187e+00 - (-7.877120e+00) = 0.338933, isn't that considered a large
difference to the point where it's safe to say that these two values are
not similar?

Another example is the t-values for x1 and x2:
x1 = 8.799994e+01   -  5.509361e+02 = -462.9362
x2 = 8.508606e+01   -  5.287634e+02 = -443.6773

An more alarming case that unfortunately I cannot share due to the data
being sensitive is that when the x variables' positions are switched, the
p-values are not the same. The p-value for a particular external regressor
went from 0 to 0.4385.

I will attempt to re-create a separate generic dataset that is similar to
the sensitive data that I am using.


On Sun, Aug 19, 2018 at 10:06 PM, alexios galanos <[hidden email]>
wrote:

> I run the code you provided and obtain the following results related to
> the external parameters:
>
>
> Case 1 (x1,x2)
> # x2 is second
>
>             Estimate   Std. Error       t value  Pr(>|t|)
> mxreg1  1.6724148 1.203377e-01  1.389767e+01 0.0000000
> mxreg2  2.5310286 1.878833e-02  1.347128e+02 0.0000000
>
> Case 2 (x2,x1)
> # i.e. x2 is now first
>
> mxreg1  2.5225382  0.04292725  58.7631024 0.000000e+00
> mxreg2  1.6782986  0.12769622  13.1428990 0.000000e+00
>
> Small differences in the coefficients are the result of the optimizer.
> There may be an issues in the
> way starting parameters are being generated based on some recent input
> from Josh Ulrich (still to investigate)
> and related to arima0 (used to generate start parameters), but otherwise
> don’t see a large problem at first glance.
>
> Alexios
>
> > On Aug 19, 2018, at 5:46 PM, GALIB KHAN <[hidden email]>
> wrote:
> >
> > Recently I have discovered a problem with a package called rugarch that
> > creates arma-garch models. The issue is that if you literally change the
> > positions of the x variables (external regressors) then you get two
> > completely different results.
> >
> > In other words:
> >
> >   - model1 = (arma(2,2) + garch(1,0) + x1 + x2)
> >   - model2 = (arma(2,2) + garch(1,0) + x2 + x1)
> >   - rugarch's output is essentially saying that model1 != model2
> >   - When the correct result should be model1 == model2
> >
> > I may not know a lot of statistics but I know for a fact that if you move
> > the x variables around, the output should still be the same.
> >
> > Am I wrong on this?
> >
> > Here's my stack exchange post that shows a generic R script proving my
> > point: Should the positioning of the external regressors change the
> output
> > of arma-garch? (Possible rugarch bug/error)
> > <https://stackoverflow.com/questions/51900177/should-the-pos
> itioning-of-the-external-regressors-change-the-output-of-arma-garc>
> >
> > Any feedback is welcomed.
> >
> > Thanks
> >
> >       [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > [hidden email] mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> > -- Subscriber-posting only. If you want to post, subscribe first.
> > -- Also note that this is not the r-help list where general R questions
> should go.
> >
>
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: rugarch gives two different results based on the same model…how is that even possible?

GALIB KHAN
In reply to this post by alexios
 Sorry for sending this again, I didn't include r-sig-finance in the email
address. I'm still adjusting in how to respond.

Alexios,

Did you set the set the seed to 1, because I'm looking at your results and
the numbers do not match with the numbers that I have provided.

I understand why the coefficients' estimates are similar but it doesn't
explain why other columns such as the t-value and pr are off by a large
margin. Also estimates for mu, ar*, ma*, omega, alpha1, and shape may have
large differences.

Take mu as an example:
-7.538187e+00 - (-7.877120e+00) = 0.338933, isn't that considered a large
difference to the point where it's safe to say that these two values are
not similar?

Another example is the t-values for x1 and x2:
x1 = 8.799994e+01   -  5.509361e+02 = -462.9362
x2 = 8.508606e+01   -  5.287634e+02 = -443.6773

An more alarming case that unfortunately I cannot share due to the data
being sensitive is that when the x variables' positions are switched, the
p-values are not the same. The p-value for a particular external regressor
went from 0 to 0.4385.

I will attempt to re-create a separate generic dataset that is similar to
the sensitive data that I am using.


Galib Khan


On Sun, Aug 19, 2018 at 10:06 PM, alexios galanos <[hidden email]>
wrote:

> I run the code you provided and obtain the following results related to
> the external parameters:
>
>
> Case 1 (x1,x2)
> # x2 is second
>
>             Estimate   Std. Error       t value  Pr(>|t|)
> mxreg1  1.6724148 1.203377e-01  1.389767e+01 0.0000000
> mxreg2  2.5310286 1.878833e-02  1.347128e+02 0.0000000
>
> Case 2 (x2,x1)
> # i.e. x2 is now first
>
> mxreg1  2.5225382  0.04292725  58.7631024 0.000000e+00
> mxreg2  1.6782986  0.12769622  13.1428990 0.000000e+00
>
> Small differences in the coefficients are the result of the optimizer.
> There may be an issues in the
> way starting parameters are being generated based on some recent input
> from Josh Ulrich (still to investigate)
> and related to arima0 (used to generate start parameters), but otherwise
> don’t see a large problem at first glance.
>
> Alexios
>
> > On Aug 19, 2018, at 5:46 PM, GALIB KHAN <[hidden email]>
> wrote:
> >
> > Recently I have discovered a problem with a package called rugarch that
> > creates arma-garch models. The issue is that if you literally change the
> > positions of the x variables (external regressors) then you get two
> > completely different results.
> >
> > In other words:
> >
> >   - model1 = (arma(2,2) + garch(1,0) + x1 + x2)
> >   - model2 = (arma(2,2) + garch(1,0) + x2 + x1)
> >   - rugarch's output is essentially saying that model1 != model2
> >   - When the correct result should be model1 == model2
> >
> > I may not know a lot of statistics but I know for a fact that if you move
> > the x variables around, the output should still be the same.
> >
> > Am I wrong on this?
> >
> > Here's my stack exchange post that shows a generic R script proving my
> > point: Should the positioning of the external regressors change the
> output
> > of arma-garch? (Possible rugarch bug/error)
> > <https://stackoverflow.com/questions/51900177/should-the-
> positioning-of-the-external-regressors-change-the-output-of-arma-garc>
> >
> > Any feedback is welcomed.
> >
> > Thanks
> >
> >       [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > [hidden email] mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> > -- Subscriber-posting only. If you want to post, subscribe first.
> > -- Also note that this is not the r-help list where general R questions
> should go.
> >
>
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: rugarch gives two different results based on the same model…how is that even possible?

alexios
I did use the seed you provided.

Use the following code for estimation:

fit <- ugarchfit(spec = spec, data = as.matrix(temp$y),solver = "nlminb", fit.control=list(scale=1))

model_maker(var1)
         Estimate  Std. Error     t value     Pr(>|t|)
mu     -7.3998577  0.69086641 -10.7109821 0.0000000000
ar1     0.3387323  0.08280162   4.0908900 0.0000429721
ar2    -0.8834201  0.06569477 -13.4473414 0.0000000000
ma1    -0.2902069  0.08598589  -3.3750525 0.0007380161
ma2     0.8660807  0.06778418  12.7770320 0.0000000000
mxreg1  1.6782992  0.12769644  13.1428825 0.0000000000
mxreg2  2.5225382  0.04292728  58.7630625 0.0000000000
omega  12.0047145  0.82986864  14.4658010 0.0000000000
alpha1  0.0000000  0.07358520   0.0000000 1.0000000000
shape  63.0103309 98.49188643   0.6397515 0.5223341761

model_maker(var2)
         Estimate  Std. Error     t value     Pr(>|t|)
mu     -7.3998549  0.69086651 -10.7109764 0.000000e+00
ar1     0.3387334  0.08280150   4.0909088 4.296861e-05
ar2    -0.8834206  0.06569433 -13.4474406 0.000000e+00
ma1    -0.2902081  0.08598562  -3.3750776 7.379487e-04
ma2     0.8660811  0.06778412  12.7770487 0.000000e+00
mxreg1  2.5225383  0.04292728  58.7630642 0.000000e+00
mxreg2  1.6782987  0.12769640  13.1428817 0.000000e+00
omega  12.0047142  0.82992363  14.4648419 0.000000e+00
alpha1  0.0000000  0.07359329   0.0000000 1.000000e+00
shape  63.0105962 98.49368444   0.6397425 5.223400e-01


I can’t see any “significant” differences, can you?
It’s completely related to the optimization/starting parameters. The “scale” is documented and not on by default (perhaps it should be).

Alexios


> On Aug 19, 2018, at 9:02 PM, GALIB KHAN <[hidden email]> wrote:
>
> Sorry for sending this again, I didn't include r-sig-finance in the email address. I'm still adjusting in how to respond.
>  
> Alexios,
>
> Did you set the set the seed to 1, because I'm looking at your results and the numbers do not match with the numbers that I have provided.
>
> I understand why the coefficients' estimates are similar but it doesn't explain why other columns such as the t-value and pr are off by a large margin. Also estimates for mu, ar*, ma*, omega, alpha1, and shape may have large differences.
>
> Take mu as an example:
> -7.538187e+00 - (-7.877120e+00) = 0.338933, isn't that considered a large difference to the point where it's safe to say that these two values are not similar?
>
> Another example is the t-values for x1 and x2:
> x1 = 8.799994e+01   -  5.509361e+02 = -462.9362
> x2 = 8.508606e+01   -  5.287634e+02 = -443.6773
>
> An more alarming case that unfortunately I cannot share due to the data being sensitive is that when the x variables' positions are switched, the p-values are not the same. The p-value for a particular external regressor went from 0 to 0.4385.
>
> I will attempt to re-create a separate generic dataset that is similar to the sensitive data that I am using.
>
>
> Galib Khan
>
>
> On Sun, Aug 19, 2018 at 10:06 PM, alexios galanos <[hidden email]> wrote:
> I run the code you provided and obtain the following results related to the external parameters:
>
>
> Case 1 (x1,x2)
> # x2 is second
>
>             Estimate   Std. Error       t value  Pr(>|t|)
> mxreg1  1.6724148 1.203377e-01  1.389767e+01 0.0000000
> mxreg2  2.5310286 1.878833e-02  1.347128e+02 0.0000000
>
> Case 2 (x2,x1)
> # i.e. x2 is now first
>
> mxreg1  2.5225382  0.04292725  58.7631024 0.000000e+00
> mxreg2  1.6782986  0.12769622  13.1428990 0.000000e+00
>
> Small differences in the coefficients are the result of the optimizer. There may be an issues in the
> way starting parameters are being generated based on some recent input from Josh Ulrich (still to investigate)
> and related to arima0 (used to generate start parameters), but otherwise don’t see a large problem at first glance.
>
> Alexios
>
> > On Aug 19, 2018, at 5:46 PM, GALIB KHAN <[hidden email]> wrote:
> >
> > Recently I have discovered a problem with a package called rugarch that
> > creates arma-garch models. The issue is that if you literally change the
> > positions of the x variables (external regressors) then you get two
> > completely different results.
> >
> > In other words:
> >
> >   - model1 = (arma(2,2) + garch(1,0) + x1 + x2)
> >   - model2 = (arma(2,2) + garch(1,0) + x2 + x1)
> >   - rugarch's output is essentially saying that model1 != model2
> >   - When the correct result should be model1 == model2
> >
> > I may not know a lot of statistics but I know for a fact that if you move
> > the x variables around, the output should still be the same.
> >
> > Am I wrong on this?
> >
> > Here's my stack exchange post that shows a generic R script proving my
> > point: Should the positioning of the external regressors change the output
> > of arma-garch? (Possible rugarch bug/error)
> > <https://stackoverflow.com/questions/51900177/should-the-positioning-of-the-external-regressors-change-the-output-of-arma-garc>
> >
> > Any feedback is welcomed.
> >
> > Thanks
> >
> >       [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > [hidden email] mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> > -- Subscriber-posting only. If you want to post, subscribe first.
> > -- Also note that this is not the r-help list where general R questions should go.
> >
>
>

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: rugarch gives two different results based on the same model…how is that even possible?

GALIB KHAN
Alexios,


Veryyyy interesting!!!!! No I cannot see any differences at all lol.

I updated the code and indeed you are correct sir. Thank you for your time
in investigating this.

I will update my stack exchange post to reflect your answer in the morning.

Again thank you for all your help!!!!

Best,
Galib Khan


On Sun, Aug 19, 2018 at 11:19 PM, alexios galanos <[hidden email]>
wrote:

> I did use the seed you provided.
>
> Use the following code for estimation:
>
> fit <- ugarchfit(spec = spec, data = as.matrix(temp$y),solver = "nlminb",
> fit.control=list(scale=1))
>
> model_maker(var1)
>          Estimate  Std. Error     t value     Pr(>|t|)
> mu     -7.3998577  0.69086641 -10.7109821 0.0000000000
> ar1     0.3387323  0.08280162   4.0908900 0.0000429721
> ar2    -0.8834201  0.06569477 -13.4473414 0.0000000000
> ma1    -0.2902069  0.08598589  -3.3750525 0.0007380161
> ma2     0.8660807  0.06778418  12.7770320 0.0000000000
> mxreg1  1.6782992  0.12769644  13.1428825 0.0000000000
> mxreg2  2.5225382  0.04292728  58.7630625 0.0000000000
> omega  12.0047145  0.82986864  14.4658010 0.0000000000
> alpha1  0.0000000  0.07358520   0.0000000 1.0000000000
> shape  63.0103309 98.49188643   0.6397515 0.5223341761
>
> model_maker(var2)
>          Estimate  Std. Error     t value     Pr(>|t|)
> mu     -7.3998549  0.69086651 -10.7109764 0.000000e+00
> ar1     0.3387334  0.08280150   4.0909088 4.296861e-05
> ar2    -0.8834206  0.06569433 -13.4474406 0.000000e+00
> ma1    -0.2902081  0.08598562  -3.3750776 7.379487e-04
> ma2     0.8660811  0.06778412  12.7770487 0.000000e+00
> mxreg1  2.5225383  0.04292728  58.7630642 0.000000e+00
> mxreg2  1.6782987  0.12769640  13.1428817 0.000000e+00
> omega  12.0047142  0.82992363  14.4648419 0.000000e+00
> alpha1  0.0000000  0.07359329   0.0000000 1.000000e+00
> shape  63.0105962 98.49368444   0.6397425 5.223400e-01
>
>
> I can’t see any “significant” differences, can you?
> It’s completely related to the optimization/starting parameters. The
> “scale” is documented and not on by default (perhaps it should be).
>
> Alexios
>
>
> > On Aug 19, 2018, at 9:02 PM, GALIB KHAN <[hidden email]>
> wrote:
> >
> > Sorry for sending this again, I didn't include r-sig-finance in the
> email address. I'm still adjusting in how to respond.
> >
> > Alexios,
> >
> > Did you set the set the seed to 1, because I'm looking at your results
> and the numbers do not match with the numbers that I have provided.
> >
> > I understand why the coefficients' estimates are similar but it doesn't
> explain why other columns such as the t-value and pr are off by a large
> margin. Also estimates for mu, ar*, ma*, omega, alpha1, and shape may have
> large differences.
> >
> > Take mu as an example:
> > -7.538187e+00 - (-7.877120e+00) = 0.338933, isn't that considered a
> large difference to the point where it's safe to say that these two values
> are not similar?
> >
> > Another example is the t-values for x1 and x2:
> > x1 = 8.799994e+01   -  5.509361e+02 = -462.9362
> > x2 = 8.508606e+01   -  5.287634e+02 = -443.6773
> >
> > An more alarming case that unfortunately I cannot share due to the data
> being sensitive is that when the x variables' positions are switched, the
> p-values are not the same. The p-value for a particular external regressor
> went from 0 to 0.4385.
> >
> > I will attempt to re-create a separate generic dataset that is similar
> to the sensitive data that I am using.
> >
> >
> > Galib Khan
> >
> >
> > On Sun, Aug 19, 2018 at 10:06 PM, alexios galanos <[hidden email]>
> wrote:
> > I run the code you provided and obtain the following results related to
> the external parameters:
> >
> >
> > Case 1 (x1,x2)
> > # x2 is second
> >
> >             Estimate   Std. Error       t value  Pr(>|t|)
> > mxreg1  1.6724148 1.203377e-01  1.389767e+01 0.0000000
> > mxreg2  2.5310286 1.878833e-02  1.347128e+02 0.0000000
> >
> > Case 2 (x2,x1)
> > # i.e. x2 is now first
> >
> > mxreg1  2.5225382  0.04292725  58.7631024 0.000000e+00
> > mxreg2  1.6782986  0.12769622  13.1428990 0.000000e+00
> >
> > Small differences in the coefficients are the result of the optimizer.
> There may be an issues in the
> > way starting parameters are being generated based on some recent input
> from Josh Ulrich (still to investigate)
> > and related to arima0 (used to generate start parameters), but otherwise
> don’t see a large problem at first glance.
> >
> > Alexios
> >
> > > On Aug 19, 2018, at 5:46 PM, GALIB KHAN <[hidden email]>
> wrote:
> > >
> > > Recently I have discovered a problem with a package called rugarch that
> > > creates arma-garch models. The issue is that if you literally change
> the
> > > positions of the x variables (external regressors) then you get two
> > > completely different results.
> > >
> > > In other words:
> > >
> > >   - model1 = (arma(2,2) + garch(1,0) + x1 + x2)
> > >   - model2 = (arma(2,2) + garch(1,0) + x2 + x1)
> > >   - rugarch's output is essentially saying that model1 != model2
> > >   - When the correct result should be model1 == model2
> > >
> > > I may not know a lot of statistics but I know for a fact that if you
> move
> > > the x variables around, the output should still be the same.
> > >
> > > Am I wrong on this?
> > >
> > > Here's my stack exchange post that shows a generic R script proving my
> > > point: Should the positioning of the external regressors change the
> output
> > > of arma-garch? (Possible rugarch bug/error)
> > > <https://stackoverflow.com/questions/51900177/should-the-
> positioning-of-the-external-regressors-change-the-output-of-arma-garc>
> > >
> > > Any feedback is welcomed.
> > >
> > > Thanks
> > >
> > >       [[alternative HTML version deleted]]
> > >
> > > _______________________________________________
> > > [hidden email] mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> > > -- Subscriber-posting only. If you want to post, subscribe first.
> > > -- Also note that this is not the r-help list where general R
> questions should go.
> > >
> >
> >
>
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: rugarch gives two different results based on the same model…how is that even possible?

GALIB KHAN
Alexios,

I changed the solver to "hybrid" but kept the scaling that you provided and
got different results when switching the positions for the external
regressors.

ugarchfit(spec = spec, data = as.matrix(temp$y),solver =
"hybrid",fit.control=list(scale=1))

So it looks like you have to change the solver to "nlminb" and update
fit.control with the scaling that you provided.

Galib


On Sun, Aug 19, 2018 at 11:41 PM, GALIB KHAN <[hidden email]>
wrote:

> Alexios,
>
>
> Veryyyy interesting!!!!! No I cannot see any differences at all lol.
>
> I updated the code and indeed you are correct sir. Thank you for your time
> in investigating this.
>
> I will update my stack exchange post to reflect your answer in the morning.
>
> Again thank you for all your help!!!!
>
> Best,
> Galib Khan
>
>
> On Sun, Aug 19, 2018 at 11:19 PM, alexios galanos <[hidden email]>
> wrote:
>
>> I did use the seed you provided.
>>
>> Use the following code for estimation:
>>
>> fit <- ugarchfit(spec = spec, data = as.matrix(temp$y),solver = "nlminb",
>> fit.control=list(scale=1))
>>
>> model_maker(var1)
>>          Estimate  Std. Error     t value     Pr(>|t|)
>> mu     -7.3998577  0.69086641 -10.7109821 0.0000000000
>> ar1     0.3387323  0.08280162   4.0908900 0.0000429721
>> ar2    -0.8834201  0.06569477 -13.4473414 0.0000000000
>> ma1    -0.2902069  0.08598589  -3.3750525 0.0007380161
>> ma2     0.8660807  0.06778418  12.7770320 0.0000000000
>> mxreg1  1.6782992  0.12769644  13.1428825 0.0000000000
>> mxreg2  2.5225382  0.04292728  58.7630625 0.0000000000
>> omega  12.0047145  0.82986864  14.4658010 0.0000000000
>> alpha1  0.0000000  0.07358520   0.0000000 1.0000000000
>> shape  63.0103309 98.49188643   0.6397515 0.5223341761
>>
>> model_maker(var2)
>>          Estimate  Std. Error     t value     Pr(>|t|)
>> mu     -7.3998549  0.69086651 -10.7109764 0.000000e+00
>> ar1     0.3387334  0.08280150   4.0909088 4.296861e-05
>> ar2    -0.8834206  0.06569433 -13.4474406 0.000000e+00
>> ma1    -0.2902081  0.08598562  -3.3750776 7.379487e-04
>> ma2     0.8660811  0.06778412  12.7770487 0.000000e+00
>> mxreg1  2.5225383  0.04292728  58.7630642 0.000000e+00
>> mxreg2  1.6782987  0.12769640  13.1428817 0.000000e+00
>> omega  12.0047142  0.82992363  14.4648419 0.000000e+00
>> alpha1  0.0000000  0.07359329   0.0000000 1.000000e+00
>> shape  63.0105962 98.49368444   0.6397425 5.223400e-01
>>
>>
>> I can’t see any “significant” differences, can you?
>> It’s completely related to the optimization/starting parameters. The
>> “scale” is documented and not on by default (perhaps it should be).
>>
>> Alexios
>>
>>
>> > On Aug 19, 2018, at 9:02 PM, GALIB KHAN <[hidden email]>
>> wrote:
>> >
>> > Sorry for sending this again, I didn't include r-sig-finance in the
>> email address. I'm still adjusting in how to respond.
>> >
>> > Alexios,
>> >
>> > Did you set the set the seed to 1, because I'm looking at your results
>> and the numbers do not match with the numbers that I have provided.
>> >
>> > I understand why the coefficients' estimates are similar but it doesn't
>> explain why other columns such as the t-value and pr are off by a large
>> margin. Also estimates for mu, ar*, ma*, omega, alpha1, and shape may have
>> large differences.
>> >
>> > Take mu as an example:
>> > -7.538187e+00 - (-7.877120e+00) = 0.338933, isn't that considered a
>> large difference to the point where it's safe to say that these two values
>> are not similar?
>> >
>> > Another example is the t-values for x1 and x2:
>> > x1 = 8.799994e+01   -  5.509361e+02 = -462.9362
>> > x2 = 8.508606e+01   -  5.287634e+02 = -443.6773
>> >
>> > An more alarming case that unfortunately I cannot share due to the data
>> being sensitive is that when the x variables' positions are switched, the
>> p-values are not the same. The p-value for a particular external regressor
>> went from 0 to 0.4385.
>> >
>> > I will attempt to re-create a separate generic dataset that is similar
>> to the sensitive data that I am using.
>> >
>> >
>> > Galib Khan
>> >
>> >
>> > On Sun, Aug 19, 2018 at 10:06 PM, alexios galanos <[hidden email]>
>> wrote:
>> > I run the code you provided and obtain the following results related to
>> the external parameters:
>> >
>> >
>> > Case 1 (x1,x2)
>> > # x2 is second
>> >
>> >             Estimate   Std. Error       t value  Pr(>|t|)
>> > mxreg1  1.6724148 1.203377e-01  1.389767e+01 0.0000000
>> > mxreg2  2.5310286 1.878833e-02  1.347128e+02 0.0000000
>> >
>> > Case 2 (x2,x1)
>> > # i.e. x2 is now first
>> >
>> > mxreg1  2.5225382  0.04292725  58.7631024 0.000000e+00
>> > mxreg2  1.6782986  0.12769622  13.1428990 0.000000e+00
>> >
>> > Small differences in the coefficients are the result of the optimizer.
>> There may be an issues in the
>> > way starting parameters are being generated based on some recent input
>> from Josh Ulrich (still to investigate)
>> > and related to arima0 (used to generate start parameters), but
>> otherwise don’t see a large problem at first glance.
>> >
>> > Alexios
>> >
>> > > On Aug 19, 2018, at 5:46 PM, GALIB KHAN <
>> [hidden email]> wrote:
>> > >
>> > > Recently I have discovered a problem with a package called rugarch
>> that
>> > > creates arma-garch models. The issue is that if you literally change
>> the
>> > > positions of the x variables (external regressors) then you get two
>> > > completely different results.
>> > >
>> > > In other words:
>> > >
>> > >   - model1 = (arma(2,2) + garch(1,0) + x1 + x2)
>> > >   - model2 = (arma(2,2) + garch(1,0) + x2 + x1)
>> > >   - rugarch's output is essentially saying that model1 != model2
>> > >   - When the correct result should be model1 == model2
>> > >
>> > > I may not know a lot of statistics but I know for a fact that if you
>> move
>> > > the x variables around, the output should still be the same.
>> > >
>> > > Am I wrong on this?
>> > >
>> > > Here's my stack exchange post that shows a generic R script proving my
>> > > point: Should the positioning of the external regressors change the
>> output
>> > > of arma-garch? (Possible rugarch bug/error)
>> > > <https://stackoverflow.com/questions/51900177/should-the-pos
>> itioning-of-the-external-regressors-change-the-output-of-arma-garc>
>> > >
>> > > Any feedback is welcomed.
>> > >
>> > > Thanks
>> > >
>> > >       [[alternative HTML version deleted]]
>> > >
>> > > _______________________________________________
>> > > [hidden email] mailing list
>> > > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>> > > -- Subscriber-posting only. If you want to post, subscribe first.
>> > > -- Also note that this is not the r-help list where general R
>> questions should go.
>> > >
>> >
>> >
>>
>>
>

Galib Khan


On Sun, Aug 19, 2018 at 11:41 PM, GALIB KHAN <[hidden email]>
wrote:

> Alexios,
>
>
> Veryyyy interesting!!!!! No I cannot see any differences at all lol.
>
> I updated the code and indeed you are correct sir. Thank you for your time
> in investigating this.
>
> I will update my stack exchange post to reflect your answer in the morning.
>
> Again thank you for all your help!!!!
>
> Best,
> Galib Khan
>
>
> On Sun, Aug 19, 2018 at 11:19 PM, alexios galanos <[hidden email]>
> wrote:
>
>> I did use the seed you provided.
>>
>> Use the following code for estimation:
>>
>> fit <- ugarchfit(spec = spec, data = as.matrix(temp$y),solver = "nlminb",
>> fit.control=list(scale=1))
>>
>> model_maker(var1)
>>          Estimate  Std. Error     t value     Pr(>|t|)
>> mu     -7.3998577  0.69086641 -10.7109821 0.0000000000
>> ar1     0.3387323  0.08280162   4.0908900 0.0000429721
>> ar2    -0.8834201  0.06569477 -13.4473414 0.0000000000
>> ma1    -0.2902069  0.08598589  -3.3750525 0.0007380161
>> ma2     0.8660807  0.06778418  12.7770320 0.0000000000
>> mxreg1  1.6782992  0.12769644  13.1428825 0.0000000000
>> mxreg2  2.5225382  0.04292728  58.7630625 0.0000000000
>> omega  12.0047145  0.82986864  14.4658010 0.0000000000
>> alpha1  0.0000000  0.07358520   0.0000000 1.0000000000
>> shape  63.0103309 98.49188643   0.6397515 0.5223341761
>>
>> model_maker(var2)
>>          Estimate  Std. Error     t value     Pr(>|t|)
>> mu     -7.3998549  0.69086651 -10.7109764 0.000000e+00
>> ar1     0.3387334  0.08280150   4.0909088 4.296861e-05
>> ar2    -0.8834206  0.06569433 -13.4474406 0.000000e+00
>> ma1    -0.2902081  0.08598562  -3.3750776 7.379487e-04
>> ma2     0.8660811  0.06778412  12.7770487 0.000000e+00
>> mxreg1  2.5225383  0.04292728  58.7630642 0.000000e+00
>> mxreg2  1.6782987  0.12769640  13.1428817 0.000000e+00
>> omega  12.0047142  0.82992363  14.4648419 0.000000e+00
>> alpha1  0.0000000  0.07359329   0.0000000 1.000000e+00
>> shape  63.0105962 98.49368444   0.6397425 5.223400e-01
>>
>>
>> I can’t see any “significant” differences, can you?
>> It’s completely related to the optimization/starting parameters. The
>> “scale” is documented and not on by default (perhaps it should be).
>>
>> Alexios
>>
>>
>> > On Aug 19, 2018, at 9:02 PM, GALIB KHAN <[hidden email]>
>> wrote:
>> >
>> > Sorry for sending this again, I didn't include r-sig-finance in the
>> email address. I'm still adjusting in how to respond.
>> >
>> > Alexios,
>> >
>> > Did you set the set the seed to 1, because I'm looking at your results
>> and the numbers do not match with the numbers that I have provided.
>> >
>> > I understand why the coefficients' estimates are similar but it doesn't
>> explain why other columns such as the t-value and pr are off by a large
>> margin. Also estimates for mu, ar*, ma*, omega, alpha1, and shape may have
>> large differences.
>> >
>> > Take mu as an example:
>> > -7.538187e+00 - (-7.877120e+00) = 0.338933, isn't that considered a
>> large difference to the point where it's safe to say that these two values
>> are not similar?
>> >
>> > Another example is the t-values for x1 and x2:
>> > x1 = 8.799994e+01   -  5.509361e+02 = -462.9362
>> > x2 = 8.508606e+01   -  5.287634e+02 = -443.6773
>> >
>> > An more alarming case that unfortunately I cannot share due to the data
>> being sensitive is that when the x variables' positions are switched, the
>> p-values are not the same. The p-value for a particular external regressor
>> went from 0 to 0.4385.
>> >
>> > I will attempt to re-create a separate generic dataset that is similar
>> to the sensitive data that I am using.
>> >
>> >
>> > Galib Khan
>> >
>> >
>> > On Sun, Aug 19, 2018 at 10:06 PM, alexios galanos <[hidden email]>
>> wrote:
>> > I run the code you provided and obtain the following results related to
>> the external parameters:
>> >
>> >
>> > Case 1 (x1,x2)
>> > # x2 is second
>> >
>> >             Estimate   Std. Error       t value  Pr(>|t|)
>> > mxreg1  1.6724148 1.203377e-01  1.389767e+01 0.0000000
>> > mxreg2  2.5310286 1.878833e-02  1.347128e+02 0.0000000
>> >
>> > Case 2 (x2,x1)
>> > # i.e. x2 is now first
>> >
>> > mxreg1  2.5225382  0.04292725  58.7631024 0.000000e+00
>> > mxreg2  1.6782986  0.12769622  13.1428990 0.000000e+00
>> >
>> > Small differences in the coefficients are the result of the optimizer.
>> There may be an issues in the
>> > way starting parameters are being generated based on some recent input
>> from Josh Ulrich (still to investigate)
>> > and related to arima0 (used to generate start parameters), but
>> otherwise don’t see a large problem at first glance.
>> >
>> > Alexios
>> >
>> > > On Aug 19, 2018, at 5:46 PM, GALIB KHAN <
>> [hidden email]> wrote:
>> > >
>> > > Recently I have discovered a problem with a package called rugarch
>> that
>> > > creates arma-garch models. The issue is that if you literally change
>> the
>> > > positions of the x variables (external regressors) then you get two
>> > > completely different results.
>> > >
>> > > In other words:
>> > >
>> > >   - model1 = (arma(2,2) + garch(1,0) + x1 + x2)
>> > >   - model2 = (arma(2,2) + garch(1,0) + x2 + x1)
>> > >   - rugarch's output is essentially saying that model1 != model2
>> > >   - When the correct result should be model1 == model2
>> > >
>> > > I may not know a lot of statistics but I know for a fact that if you
>> move
>> > > the x variables around, the output should still be the same.
>> > >
>> > > Am I wrong on this?
>> > >
>> > > Here's my stack exchange post that shows a generic R script proving my
>> > > point: Should the positioning of the external regressors change the
>> output
>> > > of arma-garch? (Possible rugarch bug/error)
>> > > <https://stackoverflow.com/questions/51900177/should-the-pos
>> itioning-of-the-external-regressors-change-the-output-of-arma-garc>
>> > >
>> > > Any feedback is welcomed.
>> > >
>> > > Thanks
>> > >
>> > >       [[alternative HTML version deleted]]
>> > >
>> > > _______________________________________________
>> > > [hidden email] mailing list
>> > > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>> > > -- Subscriber-posting only. If you want to post, subscribe first.
>> > > -- Also note that this is not the r-help list where general R
>> questions should go.
>> > >
>> >
>> >
>>
>>
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: rugarch gives two different results based on the same model…how is that even possible?

GALIB KHAN
Again sorry for the repeat. I accidentally created a new post when I wanted
to reply back.

Alexios,

I updated my script to use multiple solvers and compare results.

It looks like nlminb is the most accurate solver for this generic dataset
that I am using.

I posted the results on stack exchange so its easier to view the results.

If you want, I can post the results here.

Here is the url with the updated script and results:

https://stackoverflow.com/questions/51900177/should-the-positioning-of-the-external-regressors-change-the-output-of-arma-garc/

On Mon, Aug 20, 2018, 7:29 AM GALIB KHAN <[hidden email]>
wrote:

> Alexios,
>
> I changed the solver to "hybrid" but kept the scaling that you provided
> and got different results when switching the positions for the external
> regressors.
>
> ugarchfit(spec = spec, data = as.matrix(temp$y),solver =
> "hybrid",fit.control=list(scale=1))
>
> So it looks like you have to change the solver to "nlminb" and update
> fit.control with the scaling that you provided.
>
> Galib
>
>
> On Sun, Aug 19, 2018 at 11:41 PM, GALIB KHAN <
> [hidden email]> wrote:
>
>> Alexios,
>>
>>
>> Veryyyy interesting!!!!! No I cannot see any differences at all lol.
>>
>> I updated the code and indeed you are correct sir. Thank you for your
>> time in investigating this.
>>
>> I will update my stack exchange post to reflect your answer in the
>> morning.
>>
>> Again thank you for all your help!!!!
>>
>> Best,
>> Galib Khan
>>
>>
>> On Sun, Aug 19, 2018 at 11:19 PM, alexios galanos <[hidden email]>
>> wrote:
>>
>>> I did use the seed you provided.
>>>
>>> Use the following code for estimation:
>>>
>>> fit <- ugarchfit(spec = spec, data = as.matrix(temp$y),solver =
>>> "nlminb", fit.control=list(scale=1))
>>>
>>> model_maker(var1)
>>>          Estimate  Std. Error     t value     Pr(>|t|)
>>> mu     -7.3998577  0.69086641 -10.7109821 0.0000000000
>>> ar1     0.3387323  0.08280162   4.0908900 0.0000429721
>>> ar2    -0.8834201  0.06569477 -13.4473414 0.0000000000
>>> ma1    -0.2902069  0.08598589  -3.3750525 0.0007380161
>>> ma2     0.8660807  0.06778418  12.7770320 0.0000000000
>>> mxreg1  1.6782992  0.12769644  13.1428825 0.0000000000
>>> mxreg2  2.5225382  0.04292728  58.7630625 0.0000000000
>>> omega  12.0047145  0.82986864  14.4658010 0.0000000000
>>> alpha1  0.0000000  0.07358520   0.0000000 1.0000000000
>>> shape  63.0103309 98.49188643   0.6397515 0.5223341761
>>>
>>> model_maker(var2)
>>>          Estimate  Std. Error     t value     Pr(>|t|)
>>> mu     -7.3998549  0.69086651 -10.7109764 0.000000e+00
>>> ar1     0.3387334  0.08280150   4.0909088 4.296861e-05
>>> ar2    -0.8834206  0.06569433 -13.4474406 0.000000e+00
>>> ma1    -0.2902081  0.08598562  -3.3750776 7.379487e-04
>>> ma2     0.8660811  0.06778412  12.7770487 0.000000e+00
>>> mxreg1  2.5225383  0.04292728  58.7630642 0.000000e+00
>>> mxreg2  1.6782987  0.12769640  13.1428817 0.000000e+00
>>> omega  12.0047142  0.82992363  14.4648419 0.000000e+00
>>> alpha1  0.0000000  0.07359329   0.0000000 1.000000e+00
>>> shape  63.0105962 98.49368444   0.6397425 5.223400e-01
>>>
>>>
>>> I can’t see any “significant” differences, can you?
>>> It’s completely related to the optimization/starting parameters. The
>>> “scale” is documented and not on by default (perhaps it should be).
>>>
>>> Alexios
>>>
>>>
>>> > On Aug 19, 2018, at 9:02 PM, GALIB KHAN <[hidden email]>
>>> wrote:
>>> >
>>> > Sorry for sending this again, I didn't include r-sig-finance in the
>>> email address. I'm still adjusting in how to respond.
>>> >
>>> > Alexios,
>>> >
>>> > Did you set the set the seed to 1, because I'm looking at your results
>>> and the numbers do not match with the numbers that I have provided.
>>> >
>>> > I understand why the coefficients' estimates are similar but it
>>> doesn't explain why other columns such as the t-value and pr are off by a
>>> large margin. Also estimates for mu, ar*, ma*, omega, alpha1, and shape may
>>> have large differences.
>>> >
>>> > Take mu as an example:
>>> > -7.538187e+00 - (-7.877120e+00) = 0.338933, isn't that considered a
>>> large difference to the point where it's safe to say that these two values
>>> are not similar?
>>> >
>>> > Another example is the t-values for x1 and x2:
>>> > x1 = 8.799994e+01   -  5.509361e+02 = -462.9362
>>> > x2 = 8.508606e+01   -  5.287634e+02 = -443.6773
>>> >
>>> > An more alarming case that unfortunately I cannot share due to the
>>> data being sensitive is that when the x variables' positions are switched,
>>> the p-values are not the same. The p-value for a particular external
>>> regressor went from 0 to 0.4385.
>>> >
>>> > I will attempt to re-create a separate generic dataset that is similar
>>> to the sensitive data that I am using.
>>> >
>>> >
>>> > Galib Khan
>>> >
>>> >
>>> > On Sun, Aug 19, 2018 at 10:06 PM, alexios galanos <[hidden email]>
>>> wrote:
>>> > I run the code you provided and obtain the following results related
>>> to the external parameters:
>>> >
>>> >
>>> > Case 1 (x1,x2)
>>> > # x2 is second
>>> >
>>> >             Estimate   Std. Error       t value  Pr(>|t|)
>>> > mxreg1  1.6724148 1.203377e-01  1.389767e+01 0.0000000
>>> > mxreg2  2.5310286 1.878833e-02  1.347128e+02 0.0000000
>>> >
>>> > Case 2 (x2,x1)
>>> > # i.e. x2 is now first
>>> >
>>> > mxreg1  2.5225382  0.04292725  58.7631024 0.000000e+00
>>> > mxreg2  1.6782986  0.12769622  13.1428990 0.000000e+00
>>> >
>>> > Small differences in the coefficients are the result of the optimizer.
>>> There may be an issues in the
>>> > way starting parameters are being generated based on some recent input
>>> from Josh Ulrich (still to investigate)
>>> > and related to arima0 (used to generate start parameters), but
>>> otherwise don’t see a large problem at first glance.
>>> >
>>> > Alexios
>>> >
>>> > > On Aug 19, 2018, at 5:46 PM, GALIB KHAN <
>>> [hidden email]> wrote:
>>> > >
>>> > > Recently I have discovered a problem with a package called rugarch
>>> that
>>> > > creates arma-garch models. The issue is that if you literally change
>>> the
>>> > > positions of the x variables (external regressors) then you get two
>>> > > completely different results.
>>> > >
>>> > > In other words:
>>> > >
>>> > >   - model1 = (arma(2,2) + garch(1,0) + x1 + x2)
>>> > >   - model2 = (arma(2,2) + garch(1,0) + x2 + x1)
>>> > >   - rugarch's output is essentially saying that model1 != model2
>>> > >   - When the correct result should be model1 == model2
>>> > >
>>> > > I may not know a lot of statistics but I know for a fact that if you
>>> move
>>> > > the x variables around, the output should still be the same.
>>> > >
>>> > > Am I wrong on this?
>>> > >
>>> > > Here's my stack exchange post that shows a generic R script proving
>>> my
>>> > > point: Should the positioning of the external regressors change the
>>> output
>>> > > of arma-garch? (Possible rugarch bug/error)
>>> > > <
>>> https://stackoverflow.com/questions/51900177/should-the-positioning-of-the-external-regressors-change-the-output-of-arma-garc
>>> >
>>> > >
>>> > > Any feedback is welcomed.
>>> > >
>>> > > Thanks
>>> > >
>>> > >       [[alternative HTML version deleted]]
>>> > >
>>> > > _______________________________________________
>>> > > [hidden email] mailing list
>>> > > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>>> > > -- Subscriber-posting only. If you want to post, subscribe first.
>>> > > -- Also note that this is not the r-help list where general R
>>> questions should go.
>>> > >
>>> >
>>> >
>>>
>>>
>>
>
> Galib Khan
>
>
> On Sun, Aug 19, 2018 at 11:41 PM, GALIB KHAN <
> [hidden email]> wrote:
>
>> Alexios,
>>
>>
>> Veryyyy interesting!!!!! No I cannot see any differences at all lol.
>>
>> I updated the code and indeed you are correct sir. Thank you for your
>> time in investigating this.
>>
>> I will update my stack exchange post to reflect your answer in the
>> morning.
>>
>> Again thank you for all your help!!!!
>>
>> Best,
>> Galib Khan
>>
>>
>> On Sun, Aug 19, 2018 at 11:19 PM, alexios galanos <[hidden email]>
>> wrote:
>>
>>> I did use the seed you provided.
>>>
>>> Use the following code for estimation:
>>>
>>> fit <- ugarchfit(spec = spec, data = as.matrix(temp$y),solver =
>>> "nlminb", fit.control=list(scale=1))
>>>
>>> model_maker(var1)
>>>          Estimate  Std. Error     t value     Pr(>|t|)
>>> mu     -7.3998577  0.69086641 -10.7109821 0.0000000000
>>> ar1     0.3387323  0.08280162   4.0908900 0.0000429721
>>> ar2    -0.8834201  0.06569477 -13.4473414 0.0000000000
>>> ma1    -0.2902069  0.08598589  -3.3750525 0.0007380161
>>> ma2     0.8660807  0.06778418  12.7770320 0.0000000000
>>> mxreg1  1.6782992  0.12769644  13.1428825 0.0000000000
>>> mxreg2  2.5225382  0.04292728  58.7630625 0.0000000000
>>> omega  12.0047145  0.82986864  14.4658010 0.0000000000
>>> alpha1  0.0000000  0.07358520   0.0000000 1.0000000000
>>> shape  63.0103309 98.49188643   0.6397515 0.5223341761
>>>
>>> model_maker(var2)
>>>          Estimate  Std. Error     t value     Pr(>|t|)
>>> mu     -7.3998549  0.69086651 -10.7109764 0.000000e+00
>>> ar1     0.3387334  0.08280150   4.0909088 4.296861e-05
>>> ar2    -0.8834206  0.06569433 -13.4474406 0.000000e+00
>>> ma1    -0.2902081  0.08598562  -3.3750776 7.379487e-04
>>> ma2     0.8660811  0.06778412  12.7770487 0.000000e+00
>>> mxreg1  2.5225383  0.04292728  58.7630642 0.000000e+00
>>> mxreg2  1.6782987  0.12769640  13.1428817 0.000000e+00
>>> omega  12.0047142  0.82992363  14.4648419 0.000000e+00
>>> alpha1  0.0000000  0.07359329   0.0000000 1.000000e+00
>>> shape  63.0105962 98.49368444   0.6397425 5.223400e-01
>>>
>>>
>>> I can’t see any “significant” differences, can you?
>>> It’s completely related to the optimization/starting parameters. The
>>> “scale” is documented and not on by default (perhaps it should be).
>>>
>>> Alexios
>>>
>>>
>>> > On Aug 19, 2018, at 9:02 PM, GALIB KHAN <[hidden email]>
>>> wrote:
>>> >
>>> > Sorry for sending this again, I didn't include r-sig-finance in the
>>> email address. I'm still adjusting in how to respond.
>>> >
>>> > Alexios,
>>> >
>>> > Did you set the set the seed to 1, because I'm looking at your results
>>> and the numbers do not match with the numbers that I have provided.
>>> >
>>> > I understand why the coefficients' estimates are similar but it
>>> doesn't explain why other columns such as the t-value and pr are off by a
>>> large margin. Also estimates for mu, ar*, ma*, omega, alpha1, and shape may
>>> have large differences.
>>> >
>>> > Take mu as an example:
>>> > -7.538187e+00 - (-7.877120e+00) = 0.338933, isn't that considered a
>>> large difference to the point where it's safe to say that these two values
>>> are not similar?
>>> >
>>> > Another example is the t-values for x1 and x2:
>>> > x1 = 8.799994e+01   -  5.509361e+02 = -462.9362
>>> > x2 = 8.508606e+01   -  5.287634e+02 = -443.6773
>>> >
>>> > An more alarming case that unfortunately I cannot share due to the
>>> data being sensitive is that when the x variables' positions are switched,
>>> the p-values are not the same. The p-value for a particular external
>>> regressor went from 0 to 0.4385.
>>> >
>>> > I will attempt to re-create a separate generic dataset that is similar
>>> to the sensitive data that I am using.
>>> >
>>> >
>>> > Galib Khan
>>> >
>>> >
>>> > On Sun, Aug 19, 2018 at 10:06 PM, alexios galanos <[hidden email]>
>>> wrote:
>>> > I run the code you provided and obtain the following results related
>>> to the external parameters:
>>> >
>>> >
>>> > Case 1 (x1,x2)
>>> > # x2 is second
>>> >
>>> >             Estimate   Std. Error       t value  Pr(>|t|)
>>> > mxreg1  1.6724148 1.203377e-01  1.389767e+01 0.0000000
>>> > mxreg2  2.5310286 1.878833e-02  1.347128e+02 0.0000000
>>> >
>>> > Case 2 (x2,x1)
>>> > # i.e. x2 is now first
>>> >
>>> > mxreg1  2.5225382  0.04292725  58.7631024 0.000000e+00
>>> > mxreg2  1.6782986  0.12769622  13.1428990 0.000000e+00
>>> >
>>> > Small differences in the coefficients are the result of the optimizer.
>>> There may be an issues in the
>>> > way starting parameters are being generated based on some recent input
>>> from Josh Ulrich (still to investigate)
>>> > and related to arima0 (used to generate start parameters), but
>>> otherwise don’t see a large problem at first glance.
>>> >
>>> > Alexios
>>> >
>>> > > On Aug 19, 2018, at 5:46 PM, GALIB KHAN <
>>> [hidden email]> wrote:
>>> > >
>>> > > Recently I have discovered a problem with a package called rugarch
>>> that
>>> > > creates arma-garch models. The issue is that if you literally change
>>> the
>>> > > positions of the x variables (external regressors) then you get two
>>> > > completely different results.
>>> > >
>>> > > In other words:
>>> > >
>>> > >   - model1 = (arma(2,2) + garch(1,0) + x1 + x2)
>>> > >   - model2 = (arma(2,2) + garch(1,0) + x2 + x1)
>>> > >   - rugarch's output is essentially saying that model1 != model2
>>> > >   - When the correct result should be model1 == model2
>>> > >
>>> > > I may not know a lot of statistics but I know for a fact that if you
>>> move
>>> > > the x variables around, the output should still be the same.
>>> > >
>>> > > Am I wrong on this?
>>> > >
>>> > > Here's my stack exchange post that shows a generic R script proving
>>> my
>>> > > point: Should the positioning of the external regressors change the
>>> output
>>> > > of arma-garch? (Possible rugarch bug/error)
>>> > > <
>>> https://stackoverflow.com/questions/51900177/should-the-positioning-of-the-external-regressors-change-the-output-of-arma-garc
>>> >
>>> > >
>>> > > Any feedback is welcomed.
>>> > >
>>> > > Thanks
>>> > >
>>> > >       [[alternative HTML version deleted]]
>>> > >
>>> > > _______________________________________________
>>> > > [hidden email] mailing list
>>> > > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>>> > > -- Subscriber-posting only. If you want to post, subscribe first.
>>> > > -- Also note that this is not the r-help list where general R
>>> questions should go.
>>> > >
>>> >
>>> >
>>>
>>>
>>
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: rugarch gives two different results based on the same model…how is that even possible?

GALIB KHAN
Read your documentation and a post describing the details of fit.control
and solver.control

Let mess around those parameters and I'll report my findings back to you.


On Mon, Aug 20, 2018, 9:24 PM GALIB KHAN <[hidden email]>
wrote:

> Again sorry for the repeat. I accidentally created a new post when I
> wanted to reply back.
>
> Alexios,
>
> I updated my script to use multiple solvers and compare results.
>
> It looks like nlminb is the most accurate solver for this generic dataset
> that I am using.
>
> I posted the results on stack exchange so its easier to view the results.
>
> If you want, I can post the results here.
>
> Here is the url with the updated script and results:
>
>
> https://stackoverflow.com/questions/51900177/should-the-positioning-of-the-external-regressors-change-the-output-of-arma-garc/
>
> On Mon, Aug 20, 2018, 7:29 AM GALIB KHAN <[hidden email]>
> wrote:
>
>> Alexios,
>>
>> I changed the solver to "hybrid" but kept the scaling that you provided
>> and got different results when switching the positions for the external
>> regressors.
>>
>> ugarchfit(spec = spec, data = as.matrix(temp$y),solver =
>> "hybrid",fit.control=list(scale=1))
>>
>> So it looks like you have to change the solver to "nlminb" and update
>> fit.control with the scaling that you provided.
>>
>> Galib
>>
>>
>> On Sun, Aug 19, 2018 at 11:41 PM, GALIB KHAN <
>> [hidden email]> wrote:
>>
>>> Alexios,
>>>
>>>
>>> Veryyyy interesting!!!!! No I cannot see any differences at all lol.
>>>
>>> I updated the code and indeed you are correct sir. Thank you for your
>>> time in investigating this.
>>>
>>> I will update my stack exchange post to reflect your answer in the
>>> morning.
>>>
>>> Again thank you for all your help!!!!
>>>
>>> Best,
>>> Galib Khan
>>>
>>>
>>> On Sun, Aug 19, 2018 at 11:19 PM, alexios galanos <[hidden email]>
>>> wrote:
>>>
>>>> I did use the seed you provided.
>>>>
>>>> Use the following code for estimation:
>>>>
>>>> fit <- ugarchfit(spec = spec, data = as.matrix(temp$y),solver =
>>>> "nlminb", fit.control=list(scale=1))
>>>>
>>>> model_maker(var1)
>>>>          Estimate  Std. Error     t value     Pr(>|t|)
>>>> mu     -7.3998577  0.69086641 -10.7109821 0.0000000000
>>>> ar1     0.3387323  0.08280162   4.0908900 0.0000429721
>>>> ar2    -0.8834201  0.06569477 -13.4473414 0.0000000000
>>>> ma1    -0.2902069  0.08598589  -3.3750525 0.0007380161
>>>> ma2     0.8660807  0.06778418  12.7770320 0.0000000000
>>>> mxreg1  1.6782992  0.12769644  13.1428825 0.0000000000
>>>> mxreg2  2.5225382  0.04292728  58.7630625 0.0000000000
>>>> omega  12.0047145  0.82986864  14.4658010 0.0000000000
>>>> alpha1  0.0000000  0.07358520   0.0000000 1.0000000000
>>>> shape  63.0103309 98.49188643   0.6397515 0.5223341761
>>>>
>>>> model_maker(var2)
>>>>          Estimate  Std. Error     t value     Pr(>|t|)
>>>> mu     -7.3998549  0.69086651 -10.7109764 0.000000e+00
>>>> ar1     0.3387334  0.08280150   4.0909088 4.296861e-05
>>>> ar2    -0.8834206  0.06569433 -13.4474406 0.000000e+00
>>>> ma1    -0.2902081  0.08598562  -3.3750776 7.379487e-04
>>>> ma2     0.8660811  0.06778412  12.7770487 0.000000e+00
>>>> mxreg1  2.5225383  0.04292728  58.7630642 0.000000e+00
>>>> mxreg2  1.6782987  0.12769640  13.1428817 0.000000e+00
>>>> omega  12.0047142  0.82992363  14.4648419 0.000000e+00
>>>> alpha1  0.0000000  0.07359329   0.0000000 1.000000e+00
>>>> shape  63.0105962 98.49368444   0.6397425 5.223400e-01
>>>>
>>>>
>>>> I can’t see any “significant” differences, can you?
>>>> It’s completely related to the optimization/starting parameters. The
>>>> “scale” is documented and not on by default (perhaps it should be).
>>>>
>>>> Alexios
>>>>
>>>>
>>>> > On Aug 19, 2018, at 9:02 PM, GALIB KHAN <
>>>> [hidden email]> wrote:
>>>> >
>>>> > Sorry for sending this again, I didn't include r-sig-finance in the
>>>> email address. I'm still adjusting in how to respond.
>>>> >
>>>> > Alexios,
>>>> >
>>>> > Did you set the set the seed to 1, because I'm looking at your
>>>> results and the numbers do not match with the numbers that I have provided.
>>>> >
>>>> > I understand why the coefficients' estimates are similar but it
>>>> doesn't explain why other columns such as the t-value and pr are off by a
>>>> large margin. Also estimates for mu, ar*, ma*, omega, alpha1, and shape may
>>>> have large differences.
>>>> >
>>>> > Take mu as an example:
>>>> > -7.538187e+00 - (-7.877120e+00) = 0.338933, isn't that considered a
>>>> large difference to the point where it's safe to say that these two values
>>>> are not similar?
>>>> >
>>>> > Another example is the t-values for x1 and x2:
>>>> > x1 = 8.799994e+01   -  5.509361e+02 = -462.9362
>>>> > x2 = 8.508606e+01   -  5.287634e+02 = -443.6773
>>>> >
>>>> > An more alarming case that unfortunately I cannot share due to the
>>>> data being sensitive is that when the x variables' positions are switched,
>>>> the p-values are not the same. The p-value for a particular external
>>>> regressor went from 0 to 0.4385.
>>>> >
>>>> > I will attempt to re-create a separate generic dataset that is
>>>> similar to the sensitive data that I am using.
>>>> >
>>>> >
>>>> > Galib Khan
>>>> >
>>>> >
>>>> > On Sun, Aug 19, 2018 at 10:06 PM, alexios galanos <
>>>> [hidden email]> wrote:
>>>> > I run the code you provided and obtain the following results related
>>>> to the external parameters:
>>>> >
>>>> >
>>>> > Case 1 (x1,x2)
>>>> > # x2 is second
>>>> >
>>>> >             Estimate   Std. Error       t value  Pr(>|t|)
>>>> > mxreg1  1.6724148 1.203377e-01  1.389767e+01 0.0000000
>>>> > mxreg2  2.5310286 1.878833e-02  1.347128e+02 0.0000000
>>>> >
>>>> > Case 2 (x2,x1)
>>>> > # i.e. x2 is now first
>>>> >
>>>> > mxreg1  2.5225382  0.04292725  58.7631024 0.000000e+00
>>>> > mxreg2  1.6782986  0.12769622  13.1428990 0.000000e+00
>>>> >
>>>> > Small differences in the coefficients are the result of the
>>>> optimizer. There may be an issues in the
>>>> > way starting parameters are being generated based on some recent
>>>> input from Josh Ulrich (still to investigate)
>>>> > and related to arima0 (used to generate start parameters), but
>>>> otherwise don’t see a large problem at first glance.
>>>> >
>>>> > Alexios
>>>> >
>>>> > > On Aug 19, 2018, at 5:46 PM, GALIB KHAN <
>>>> [hidden email]> wrote:
>>>> > >
>>>> > > Recently I have discovered a problem with a package called rugarch
>>>> that
>>>> > > creates arma-garch models. The issue is that if you literally
>>>> change the
>>>> > > positions of the x variables (external regressors) then you get two
>>>> > > completely different results.
>>>> > >
>>>> > > In other words:
>>>> > >
>>>> > >   - model1 = (arma(2,2) + garch(1,0) + x1 + x2)
>>>> > >   - model2 = (arma(2,2) + garch(1,0) + x2 + x1)
>>>> > >   - rugarch's output is essentially saying that model1 != model2
>>>> > >   - When the correct result should be model1 == model2
>>>> > >
>>>> > > I may not know a lot of statistics but I know for a fact that if
>>>> you move
>>>> > > the x variables around, the output should still be the same.
>>>> > >
>>>> > > Am I wrong on this?
>>>> > >
>>>> > > Here's my stack exchange post that shows a generic R script proving
>>>> my
>>>> > > point: Should the positioning of the external regressors change the
>>>> output
>>>> > > of arma-garch? (Possible rugarch bug/error)
>>>> > > <
>>>> https://stackoverflow.com/questions/51900177/should-the-positioning-of-the-external-regressors-change-the-output-of-arma-garc
>>>> >
>>>> > >
>>>> > > Any feedback is welcomed.
>>>> > >
>>>> > > Thanks
>>>> > >
>>>> > >       [[alternative HTML version deleted]]
>>>> > >
>>>> > > _______________________________________________
>>>> > > [hidden email] mailing list
>>>> > > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>>>> > > -- Subscriber-posting only. If you want to post, subscribe first.
>>>> > > -- Also note that this is not the r-help list where general R
>>>> questions should go.
>>>> > >
>>>> >
>>>> >
>>>>
>>>>
>>>
>>
>> Galib Khan
>>
>>
>> On Sun, Aug 19, 2018 at 11:41 PM, GALIB KHAN <
>> [hidden email]> wrote:
>>
>>> Alexios,
>>>
>>>
>>> Veryyyy interesting!!!!! No I cannot see any differences at all lol.
>>>
>>> I updated the code and indeed you are correct sir. Thank you for your
>>> time in investigating this.
>>>
>>> I will update my stack exchange post to reflect your answer in the
>>> morning.
>>>
>>> Again thank you for all your help!!!!
>>>
>>> Best,
>>> Galib Khan
>>>
>>>
>>> On Sun, Aug 19, 2018 at 11:19 PM, alexios galanos <[hidden email]>
>>> wrote:
>>>
>>>> I did use the seed you provided.
>>>>
>>>> Use the following code for estimation:
>>>>
>>>> fit <- ugarchfit(spec = spec, data = as.matrix(temp$y),solver =
>>>> "nlminb", fit.control=list(scale=1))
>>>>
>>>> model_maker(var1)
>>>>          Estimate  Std. Error     t value     Pr(>|t|)
>>>> mu     -7.3998577  0.69086641 -10.7109821 0.0000000000
>>>> ar1     0.3387323  0.08280162   4.0908900 0.0000429721
>>>> ar2    -0.8834201  0.06569477 -13.4473414 0.0000000000
>>>> ma1    -0.2902069  0.08598589  -3.3750525 0.0007380161
>>>> ma2     0.8660807  0.06778418  12.7770320 0.0000000000
>>>> mxreg1  1.6782992  0.12769644  13.1428825 0.0000000000
>>>> mxreg2  2.5225382  0.04292728  58.7630625 0.0000000000
>>>> omega  12.0047145  0.82986864  14.4658010 0.0000000000
>>>> alpha1  0.0000000  0.07358520   0.0000000 1.0000000000
>>>> shape  63.0103309 98.49188643   0.6397515 0.5223341761
>>>>
>>>> model_maker(var2)
>>>>          Estimate  Std. Error     t value     Pr(>|t|)
>>>> mu     -7.3998549  0.69086651 -10.7109764 0.000000e+00
>>>> ar1     0.3387334  0.08280150   4.0909088 4.296861e-05
>>>> ar2    -0.8834206  0.06569433 -13.4474406 0.000000e+00
>>>> ma1    -0.2902081  0.08598562  -3.3750776 7.379487e-04
>>>> ma2     0.8660811  0.06778412  12.7770487 0.000000e+00
>>>> mxreg1  2.5225383  0.04292728  58.7630642 0.000000e+00
>>>> mxreg2  1.6782987  0.12769640  13.1428817 0.000000e+00
>>>> omega  12.0047142  0.82992363  14.4648419 0.000000e+00
>>>> alpha1  0.0000000  0.07359329   0.0000000 1.000000e+00
>>>> shape  63.0105962 98.49368444   0.6397425 5.223400e-01
>>>>
>>>>
>>>> I can’t see any “significant” differences, can you?
>>>> It’s completely related to the optimization/starting parameters. The
>>>> “scale” is documented and not on by default (perhaps it should be).
>>>>
>>>> Alexios
>>>>
>>>>
>>>> > On Aug 19, 2018, at 9:02 PM, GALIB KHAN <
>>>> [hidden email]> wrote:
>>>> >
>>>> > Sorry for sending this again, I didn't include r-sig-finance in the
>>>> email address. I'm still adjusting in how to respond.
>>>> >
>>>> > Alexios,
>>>> >
>>>> > Did you set the set the seed to 1, because I'm looking at your
>>>> results and the numbers do not match with the numbers that I have provided.
>>>> >
>>>> > I understand why the coefficients' estimates are similar but it
>>>> doesn't explain why other columns such as the t-value and pr are off by a
>>>> large margin. Also estimates for mu, ar*, ma*, omega, alpha1, and shape may
>>>> have large differences.
>>>> >
>>>> > Take mu as an example:
>>>> > -7.538187e+00 - (-7.877120e+00) = 0.338933, isn't that considered a
>>>> large difference to the point where it's safe to say that these two values
>>>> are not similar?
>>>> >
>>>> > Another example is the t-values for x1 and x2:
>>>> > x1 = 8.799994e+01   -  5.509361e+02 = -462.9362
>>>> > x2 = 8.508606e+01   -  5.287634e+02 = -443.6773
>>>> >
>>>> > An more alarming case that unfortunately I cannot share due to the
>>>> data being sensitive is that when the x variables' positions are switched,
>>>> the p-values are not the same. The p-value for a particular external
>>>> regressor went from 0 to 0.4385.
>>>> >
>>>> > I will attempt to re-create a separate generic dataset that is
>>>> similar to the sensitive data that I am using.
>>>> >
>>>> >
>>>> > Galib Khan
>>>> >
>>>> >
>>>> > On Sun, Aug 19, 2018 at 10:06 PM, alexios galanos <
>>>> [hidden email]> wrote:
>>>> > I run the code you provided and obtain the following results related
>>>> to the external parameters:
>>>> >
>>>> >
>>>> > Case 1 (x1,x2)
>>>> > # x2 is second
>>>> >
>>>> >             Estimate   Std. Error       t value  Pr(>|t|)
>>>> > mxreg1  1.6724148 1.203377e-01  1.389767e+01 0.0000000
>>>> > mxreg2  2.5310286 1.878833e-02  1.347128e+02 0.0000000
>>>> >
>>>> > Case 2 (x2,x1)
>>>> > # i.e. x2 is now first
>>>> >
>>>> > mxreg1  2.5225382  0.04292725  58.7631024 0.000000e+00
>>>> > mxreg2  1.6782986  0.12769622  13.1428990 0.000000e+00
>>>> >
>>>> > Small differences in the coefficients are the result of the
>>>> optimizer. There may be an issues in the
>>>> > way starting parameters are being generated based on some recent
>>>> input from Josh Ulrich (still to investigate)
>>>> > and related to arima0 (used to generate start parameters), but
>>>> otherwise don’t see a large problem at first glance.
>>>> >
>>>> > Alexios
>>>> >
>>>> > > On Aug 19, 2018, at 5:46 PM, GALIB KHAN <
>>>> [hidden email]> wrote:
>>>> > >
>>>> > > Recently I have discovered a problem with a package called rugarch
>>>> that
>>>> > > creates arma-garch models. The issue is that if you literally
>>>> change the
>>>> > > positions of the x variables (external regressors) then you get two
>>>> > > completely different results.
>>>> > >
>>>> > > In other words:
>>>> > >
>>>> > >   - model1 = (arma(2,2) + garch(1,0) + x1 + x2)
>>>> > >   - model2 = (arma(2,2) + garch(1,0) + x2 + x1)
>>>> > >   - rugarch's output is essentially saying that model1 != model2
>>>> > >   - When the correct result should be model1 == model2
>>>> > >
>>>> > > I may not know a lot of statistics but I know for a fact that if
>>>> you move
>>>> > > the x variables around, the output should still be the same.
>>>> > >
>>>> > > Am I wrong on this?
>>>> > >
>>>> > > Here's my stack exchange post that shows a generic R script proving
>>>> my
>>>> > > point: Should the positioning of the external regressors change the
>>>> output
>>>> > > of arma-garch? (Possible rugarch bug/error)
>>>> > > <
>>>> https://stackoverflow.com/questions/51900177/should-the-positioning-of-the-external-regressors-change-the-output-of-arma-garc
>>>> >
>>>> > >
>>>> > > Any feedback is welcomed.
>>>> > >
>>>> > > Thanks
>>>> > >
>>>> > >       [[alternative HTML version deleted]]
>>>> > >
>>>> > > _______________________________________________
>>>> > > [hidden email] mailing list
>>>> > > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>>>> > > -- Subscriber-posting only. If you want to post, subscribe first.
>>>> > > -- Also note that this is not the r-help list where general R
>>>> questions should go.
>>>> > >
>>>> >
>>>> >
>>>>
>>>>
>>>
>>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: rugarch gives two different results based on the same model…how is that even possible?

GALIB KHAN
In reply to this post by alexios
Alexios,

I went through your documenation for Rsolnp and I made the following
changes:

fit.control = list(scale = 0),solver.control =
list(tol=1e-20,delta=1e-20,outer.iter = 1000,inner.iter = 1000)

The sum of the total difference comes out to 4.729008e-05. Pretty darn
slick!!!!

So all is good!
Thanks for the catch!!!


On Sun, Aug 19, 2018, 11:19 PM alexios galanos <[hidden email]> wrote:

> I did use the seed you provided.
>
> Use the following code for estimation:
>
> fit <- ugarchfit(spec = spec, data = as.matrix(temp$y),solver = "nlminb",
> fit.control=list(scale=1))
>
> model_maker(var1)
>          Estimate  Std. Error     t value     Pr(>|t|)
> mu     -7.3998577  0.69086641 -10.7109821 0.0000000000
> ar1     0.3387323  0.08280162   4.0908900 0.0000429721
> ar2    -0.8834201  0.06569477 -13.4473414 0.0000000000
> ma1    -0.2902069  0.08598589  -3.3750525 0.0007380161
> ma2     0.8660807  0.06778418  12.7770320 0.0000000000
> mxreg1  1.6782992  0.12769644  13.1428825 0.0000000000
> mxreg2  2.5225382  0.04292728  58.7630625 0.0000000000
> omega  12.0047145  0.82986864  14.4658010 0.0000000000
> alpha1  0.0000000  0.07358520   0.0000000 1.0000000000
> shape  63.0103309 98.49188643   0.6397515 0.5223341761
>
> model_maker(var2)
>          Estimate  Std. Error     t value     Pr(>|t|)
> mu     -7.3998549  0.69086651 -10.7109764 0.000000e+00
> ar1     0.3387334  0.08280150   4.0909088 4.296861e-05
> ar2    -0.8834206  0.06569433 -13.4474406 0.000000e+00
> ma1    -0.2902081  0.08598562  -3.3750776 7.379487e-04
> ma2     0.8660811  0.06778412  12.7770487 0.000000e+00
> mxreg1  2.5225383  0.04292728  58.7630642 0.000000e+00
> mxreg2  1.6782987  0.12769640  13.1428817 0.000000e+00
> omega  12.0047142  0.82992363  14.4648419 0.000000e+00
> alpha1  0.0000000  0.07359329   0.0000000 1.000000e+00
> shape  63.0105962 98.49368444   0.6397425 5.223400e-01
>
>
> I can’t see any “significant” differences, can you?
> It’s completely related to the optimization/starting parameters. The
> “scale” is documented and not on by default (perhaps it should be).
>
> Alexios
>
>
> > On Aug 19, 2018, at 9:02 PM, GALIB KHAN <[hidden email]>
> wrote:
> >
> > Sorry for sending this again, I didn't include r-sig-finance in the
> email address. I'm still adjusting in how to respond.
> >
> > Alexios,
> >
> > Did you set the set the seed to 1, because I'm looking at your results
> and the numbers do not match with the numbers that I have provided.
> >
> > I understand why the coefficients' estimates are similar but it doesn't
> explain why other columns such as the t-value and pr are off by a large
> margin. Also estimates for mu, ar*, ma*, omega, alpha1, and shape may have
> large differences.
> >
> > Take mu as an example:
> > -7.538187e+00 - (-7.877120e+00) = 0.338933, isn't that considered a
> large difference to the point where it's safe to say that these two values
> are not similar?
> >
> > Another example is the t-values for x1 and x2:
> > x1 = 8.799994e+01   -  5.509361e+02 = -462.9362
> > x2 = 8.508606e+01   -  5.287634e+02 = -443.6773
> >
> > An more alarming case that unfortunately I cannot share due to the data
> being sensitive is that when the x variables' positions are switched, the
> p-values are not the same. The p-value for a particular external regressor
> went from 0 to 0.4385.
> >
> > I will attempt to re-create a separate generic dataset that is similar
> to the sensitive data that I am using.
> >
> >
> > Galib Khan
> >
> >
> > On Sun, Aug 19, 2018 at 10:06 PM, alexios galanos <[hidden email]>
> wrote:
> > I run the code you provided and obtain the following results related to
> the external parameters:
> >
> >
> > Case 1 (x1,x2)
> > # x2 is second
> >
> >             Estimate   Std. Error       t value  Pr(>|t|)
> > mxreg1  1.6724148 1.203377e-01  1.389767e+01 0.0000000
> > mxreg2  2.5310286 1.878833e-02  1.347128e+02 0.0000000
> >
> > Case 2 (x2,x1)
> > # i.e. x2 is now first
> >
> > mxreg1  2.5225382  0.04292725  58.7631024 0.000000e+00
> > mxreg2  1.6782986  0.12769622  13.1428990 0.000000e+00
> >
> > Small differences in the coefficients are the result of the optimizer.
> There may be an issues in the
> > way starting parameters are being generated based on some recent input
> from Josh Ulrich (still to investigate)
> > and related to arima0 (used to generate start parameters), but otherwise
> don’t see a large problem at first glance.
> >
> > Alexios
> >
> > > On Aug 19, 2018, at 5:46 PM, GALIB KHAN <[hidden email]>
> wrote:
> > >
> > > Recently I have discovered a problem with a package called rugarch that
> > > creates arma-garch models. The issue is that if you literally change
> the
> > > positions of the x variables (external regressors) then you get two
> > > completely different results.
> > >
> > > In other words:
> > >
> > >   - model1 = (arma(2,2) + garch(1,0) + x1 + x2)
> > >   - model2 = (arma(2,2) + garch(1,0) + x2 + x1)
> > >   - rugarch's output is essentially saying that model1 != model2
> > >   - When the correct result should be model1 == model2
> > >
> > > I may not know a lot of statistics but I know for a fact that if you
> move
> > > the x variables around, the output should still be the same.
> > >
> > > Am I wrong on this?
> > >
> > > Here's my stack exchange post that shows a generic R script proving my
> > > point: Should the positioning of the external regressors change the
> output
> > > of arma-garch? (Possible rugarch bug/error)
> > > <
> https://stackoverflow.com/questions/51900177/should-the-positioning-of-the-external-regressors-change-the-output-of-arma-garc
> >
> > >
> > > Any feedback is welcomed.
> > >
> > > Thanks
> > >
> > >       [[alternative HTML version deleted]]
> > >
> > > _______________________________________________
> > > [hidden email] mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> > > -- Subscriber-posting only. If you want to post, subscribe first.
> > > -- Also note that this is not the r-help list where general R
> questions should go.
> > >
> >
> >
>
>
On Aug 19, 2018 11:19 PM, "alexios galanos" <[hidden email]> wrote:

I did use the seed you provided.

Use the following code for estimation:

fit <- ugarchfit(spec = spec, data = as.matrix(temp$y),solver = "nlminb",
fit.control=list(scale=1))

model_maker(var1)

         Estimate  Std. Error     t value     Pr(>|t|)
mu     -7.3998577  0.69086641 -10.7109821 0.0000000000
ar1     0.3387323  0.08280162   4.0908900 0.0000429721
ar2    -0.8834201  0.06569477 -13.4473414 0.0000000000
ma1    -0.2902069  0.08598589  -3.3750525 0.0007380161
ma2     0.8660807  0.06778418  12.7770320 0.0000000000
mxreg1  1.6782992  0.12769644  13.1428825 0.0000000000
mxreg2  2.5225382  0.04292728  58.7630625 0.0000000000
omega  12.0047145  0.82986864  14.4658010 0.0000000000
alpha1  0.0000000  0.07358520   0.0000000 1.0000000000
shape  63.0103309 98.49188643   0.6397515 0.5223341761

model_maker(var2)

         Estimate  Std. Error     t value     Pr(>|t|)
mu     -7.3998549  0.69086651 -10.7109764 0.000000e+00
ar1     0.3387334  0.08280150   4.0909088 4.296861e-05
ar2    -0.8834206  0.06569433 -13.4474406 0.000000e+00
ma1    -0.2902081  0.08598562  -3.3750776 7.379487e-04
ma2     0.8660811  0.06778412  12.7770487 0.000000e+00
mxreg1  2.5225383  0.04292728  58.7630642 0.000000e+00
mxreg2  1.6782987  0.12769640  13.1428817 0.000000e+00
omega  12.0047142  0.82992363  14.4648419 0.000000e+00
alpha1  0.0000000  0.07359329   0.0000000 1.000000e+00
shape  63.0105962 98.49368444   0.6397425 5.223400e-01


I can’t see any “significant” differences, can you?
It’s completely related to the optimization/starting parameters. The
“scale” is documented and not on by default (perhaps it should be).


Alexios



> On Aug 19, 2018, at 9:02 PM, GALIB KHAN <[hidden email]>
wrote:
>
> Sorry for sending this again, I didn't include r-sig-finance in the email
address. I'm still adjusting in how to respond.
>
> Alexios,
>
> Did you set the set the seed to 1, because I'm looking at your results
and the numbers do not match with the numbers that I have provided.
>
> I understand why the coefficients' estimates are similar but it doesn't
explain why other columns such as the t-value and pr are off by a large
margin. Also estimates for mu, ar*, ma*, omega, alpha1, and shape may have
large differences.
>
> Take mu as an example:
> -7.538187e+00 - (-7.877120e+00) = 0.338933, isn't that considered a large
difference to the point where it's safe to say that these two values are
not similar?
>
> Another example is the t-values for x1 and x2:
> x1 = 8.799994e+01   -  5.509361e+02 = -462.9362
> x2 = 8.508606e+01   -  5.287634e+02 = -443.6773
>
> An more alarming case that unfortunately I cannot share due to the data
being sensitive is that when the x variables' positions are switched, the
p-values are not the same. The p-value for a particular external regressor
went from 0 to 0.4385.
>
> I will attempt to re-create a separate generic dataset that is similar to
the sensitive data that I am using.
>
>
> Galib Khan
>
>
> On Sun, Aug 19, 2018 at 10:06 PM, alexios galanos <[hidden email]>
wrote:
> I run the code you provided and obtain the following results related to
the external parameters:

>
>
> Case 1 (x1,x2)
> # x2 is second
>
>             Estimate   Std. Error       t value  Pr(>|t|)
> mxreg1  1.6724148 1.203377e-01  1.389767e+01 0.0000000
> mxreg2  2.5310286 1.878833e-02  1.347128e+02 0.0000000
>
> Case 2 (x2,x1)
> # i.e. x2 is now first
>
> mxreg1  2.5225382  0.04292725  58.7631024 0.000000e+00
> mxreg2  1.6782986  0.12769622  13.1428990 0.000000e+00
>
> Small differences in the coefficients are the result of the optimizer.
There may be an issues in the
> way starting parameters are being generated based on some recent input
from Josh Ulrich (still to investigate)
> and related to arima0 (used to generate start parameters), but otherwise
don’t see a large problem at first glance.
>
> Alexios
>
> > On Aug 19, 2018, at 5:46 PM, GALIB KHAN <[hidden email]>
wrote:

> >
> > Recently I have discovered a problem with a package called rugarch that
> > creates arma-garch models. The issue is that if you literally change the
> > positions of the x variables (external regressors) then you get two
> > completely different results.
> >
> > In other words:
> >
> >   - model1 = (arma(2,2) + garch(1,0) + x1 + x2)
> >   - model2 = (arma(2,2) + garch(1,0) + x2 + x1)
> >   - rugarch's output is essentially saying that model1 != model2
> >   - When the correct result should be model1 == model2
> >
> > I may not know a lot of statistics but I know for a fact that if you
move
> > the x variables around, the output should still be the same.
> >
> > Am I wrong on this?
> >
> > Here's my stack exchange post that shows a generic R script proving my
> > point: Should the positioning of the external regressors change the
output
> > of arma-garch? (Possible rugarch bug/error)
> > <
https://stackoverflow.com/questions/51900177/should-the-positioning-of-the-external-regressors-change-the-output-of-arma-garc

>
> >
> > Any feedback is welcomed.
> >
> > Thanks
> >
> >       [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > [hidden email] mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> > -- Subscriber-posting only. If you want to post, subscribe first.
> > -- Also note that this is not the r-help list where general R questions
should go.
> >
>
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.