Stability of trading models

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Stability of trading models

gero.schwenk
Hello together!
I've got a question regarding the assessment of stability of trading
models. Here's a short intro to the background:

I have fitted regression models with lagged independent variables to
data of daily stock returns. (The model methodology was MARS, allowing
for  second order interaction terms.) The independent variables had been
selected on basis of explorative clustering and correlation studies and
showed face validity. I used both logit and gaussian link functions and
generated out-of-sample predictions for a test window of two months (40+
observations, following the training window in time), which were of
reasonable to really satisfying quality, depending on the exact model.

In order to assess the stability of out-of-sample fit of a given model,
I would normally draw cross-validation samples and partition them into
training- and test subsets. Grounds for this would be the assumption of
independent observations contained in the model and forced onto the data
by backshifting them.

However, I'm reluctant to believe that the data-generating process
doesn't change over time, which is implied by my procedures. If this was
true and time was not an issue, it should not be necessary to
recalibrate the model, even after a long period of out-of-sample
prediction. This seems overly optimistic to me.

Returning to the question of stability assessment and cross-validation,
I would like to know if there is some pragmatic solution. Is simple
cross-validation viable? Do I need to go far into the past using some
possibly sliding training- and test-windows? Or has anybody a different
suggestion how to deal with this problem in the realm of regression models?


Kind regards,
Gero

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Stability of trading models

braverock
Gero Schwenk wrote:

> <... snip ...>
> In order to assess the stability of out-of-sample fit of a given
> model, I would normally draw cross-validation samples and partition
> them into training- and test subsets. Grounds for this would be the
> assumption of independent observations contained in the model and
> forced onto the data by backshifting them.
>
> However, I'm reluctant to believe that the data-generating process
> doesn't change over time, which is implied by my procedures. If this
> was true and time was not an issue, it should not be necessary to
> recalibrate the model, even after a long period of out-of-sample
> prediction. This seems overly optimistic to me.
>
> Returning to the question of stability assessment and
> cross-validation, I would like to know if there is some pragmatic
> solution. Is simple cross-validation viable? Do I need to go far into
> the past using some possibly sliding training- and test-windows? Or
> has anybody a different suggestion how to deal with this problem in
> the realm of regression models?
While we would all like a perfectly stable model, the reality is usually different.

Since you've said that your model is regression-based, take a look at chart.RollingRegression.  This will let you see the stability of your regression model over different rolling windows.  I suggest checking longer, shorter, and from-inception windows.

>From there, you'll have a lot of information to refine your modeling/tuning approach.

I generally am dubious that financial time series or trading models are completely stable over long periods of time.

Cheers,

    - Brian

--
Brian G. Peterson
http://braverock.com/brian/
Ph: 773-459-4973
IM: bgpbraverock

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.