Back testing

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Back testing

Bogaso
Hi,

I was looking for an idea how banks backtest their models for Expected
Shortfall. Backtesting VaR is well documented but I failed to get any
practical idea about backtesting ES.

Any pointer towards the best practice will be really helpful.

Thanks,

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Back testing

leo sea
Hi
I am also interested in in ES backtesting.
Good idea
Thanks

T�l�chargez Outlook pour Android<https://aka.ms/ghei36>

________________________________
From: R-SIG-Finance <[hidden email]> on behalf of Christofer Bogaso <[hidden email]>
Sent: Wednesday, June 10, 2020 11:38:57 AM
To: [hidden email] <[hidden email]>
Subject: [R-SIG-Finance] Back testing

Hi,

I was looking for an idea how banks backtest their models for Expected
Shortfall. Backtesting VaR is well documented but I failed to get any
practical idea about backtesting ES.

Any pointer towards the best practice will be really helpful.

Thanks,

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.

        [[alternative HTML version deleted]]


_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Back testing Expected Shortfall

braverock
In reply to this post by Bogaso
On Wed, 2020-06-10 at 15:08 +0530, Christofer Bogaso wrote:
> I was looking for an idea how banks backtest their models for
> Expected
> Shortfall. Backtesting VaR is well documented but I failed to get any
> practical idea about backtesting ES.
>
> Any pointer towards the best practice will be really helpful.

If you are using Normal VaR, then you know the Expected Shortfall
estimate too.

If you are using a different mechanism, then of course the mean loss
when the loss exceeds the VaR may be significantly different than the
Normal ES.

So, to backetesting...  the newest Basel standard replaces VaR with ES,
and requires that banks justify their use of a particular ES model that
they are using to calculate required regulatory capital.

To the best of my knowledge, the most widely used and cited approaches
are outlined here:

https://dlu-umich.github.io/docs/Research_Insight_Backtesting_Expected_Shortfall_December_2014.pdf

Generally, I like the overall methodology presented by this paper.  The
only complexity is the need to store (or be able to recalculate) the
full distribution of the tail.  I don't see this as a giant roadblock,
since the tail distribution contains additional information of interest
anyway, the shape of the tail is useful in model validation and
fitting, and disk is cheap.

The models presented in the reference above, while not to my knowledge
directly implemented in R, should be able to be constructed from data
in the recent R packages by Ardia et. al. GAS:

https://journal.r-project.org/archive/2018/RJ-2018-064/RJ-2018-064.pdf

and MSGARCH:

https://www.sciencedirect.com/science/article/pii/S0169207018300840

Regards,

Brian


--
Brian G. Peterson
ph: +1.773.459.4973
im: bgpbraverock

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Back testing Expected Shortfall

Daniel Cegiełka
śr., 10 cze 2020 o 19:23 Brian G. Peterson <[hidden email]> napisał(a):

>
> On Wed, 2020-06-10 at 15:08 +0530, Christofer Bogaso wrote:
> > I was looking for an idea how banks backtest their models for
> > Expected
> > Shortfall. Backtesting VaR is well documented but I failed to get any
> > practical idea about backtesting ES.
> >
> > Any pointer towards the best practice will be really helpful.
>
> If you are using Normal VaR, then you know the Expected Shortfall
> estimate too.
>
> If you are using a different mechanism, then of course the mean loss
> when the loss exceeds the VaR may be significantly different than the
> Normal ES.
>
> So, to backetesting...  the newest Basel standard replaces VaR with ES,
> and requires that banks justify their use of a particular ES model that
> they are using to calculate required regulatory capital.

In my opinion, there is one aspect that introduces some confusion. ES
(CVaR) is now common, but many people, perhaps out of habit, maybe for
historical reasons, still use the term VaR instead of the correct name
(ES).

Best regards,
Daniel


> Regards,
>
> Brian
>
>
> --
> Brian G. Peterson
> ph: +1.773.459.4973
> im: bgpbraverock
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions should go.

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Back testing Expected Shortfall

braverock
On Wed, 2020-06-10 at 20:08 +0200, Daniel Cegiełka wrote:

> śr., 10 cze 2020 o 19:23 Brian G. Peterson <
> [hidden email]
> > napisał(a):
> > So, to backtesting...  the newest Basel standard replaces VaR with
> > ES,
> > and requires that banks justify their use of a particular ES model
> > that
> > they are using to calculate required regulatory capital.
>
> In my opinion, there is one aspect that introduces some confusion. ES
> (CVaR) is now common, but many people, perhaps out of habit, maybe
> for
> historical reasons, still use the term VaR instead of the correct
> name
> (ES).

VaR and ES (CVaR, ETL) are mathematically related to each other, since
ES is the mean loss when the loss exceeds the VaR quantile.

Confusingly, one of the permissible tests of a bank's ES model under
Basel is the 'VaR test' which measures the number of VaR exceeding
events, and the degree of the loss eceeding VaR to evaluate whether the
*ES* model is likely valid.  This test has been widely criticized, and
should likely be avoided as anything other than a quick check of
possible suitability.

Regards,

Brian

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Back testing Expected Shortfall

alexios
In reply to this post by Daniel Cegiełka


On 6/10/20 11:08 AM, Daniel Cegiełka wrote:

> śr., 10 cze 2020 o 19:23 Brian G. Peterson <[hidden email]> napisał(a):
>>
>> On Wed, 2020-06-10 at 15:08 +0530, Christofer Bogaso wrote:
>>> I was looking for an idea how banks backtest their models for
>>> Expected
>>> Shortfall. Backtesting VaR is well documented but I failed to get any
>>> practical idea about backtesting ES.
>>>
>>> Any pointer towards the best practice will be really helpful.
>>
>> If you are using Normal VaR, then you know the Expected Shortfall
>> estimate too.
>>
>> If you are using a different mechanism, then of course the mean loss
>> when the loss exceeds the VaR may be significantly different than the
>> Normal ES.
>>
>> So, to backetesting...  the newest Basel standard replaces VaR with ES,
>> and requires that banks justify their use of a particular ES model that
>> they are using to calculate required regulatory capital.
>
> In my opinion, there is one aspect that introduces some confusion. ES
> (CVaR) is now common, but many people, perhaps out of habit, maybe for
> historical reasons, still use the term VaR instead of the correct name
> (ES).

Not sure I follow. VaR and ES are different measures. VaR is a
quantile while ES is the average loss conditional on that quantile
(i.e. the expected loss conditional that the loss is greater than
the quantile of the loss distribution).

Regards,

Alexios

>
> Best regards,
> Daniel
>
>
>> Regards,
>>
>> Brian
>>
>>
>> --
>> Brian G. Peterson
>> ph: +1.773.459.4973
>> im: bgpbraverock
>>
>> _______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>> -- Subscriber-posting only. If you want to post, subscribe first.
>> -- Also note that this is not the r-help list where general R questions should go.
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions should go.
>

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Back testing Expected Shortfall

Daniel Cegiełka
śr., 10 cze 2020 o 21:14 alexios galanos <[hidden email]> napisał(a):

>
>
>
> On 6/10/20 11:08 AM, Daniel Cegiełka wrote:
> > śr., 10 cze 2020 o 19:23 Brian G. Peterson <[hidden email]> napisał(a):
> >>
> >> On Wed, 2020-06-10 at 15:08 +0530, Christofer Bogaso wrote:
> >>> I was looking for an idea how banks backtest their models for
> >>> Expected
> >>> Shortfall. Backtesting VaR is well documented but I failed to get any
> >>> practical idea about backtesting ES.
> >>>
> >>> Any pointer towards the best practice will be really helpful.
> >>
> >> If you are using Normal VaR, then you know the Expected Shortfall
> >> estimate too.
> >>
> >> If you are using a different mechanism, then of course the mean loss
> >> when the loss exceeds the VaR may be significantly different than the
> >> Normal ES.
> >>
> >> So, to backetesting...  the newest Basel standard replaces VaR with ES,
> >> and requires that banks justify their use of a particular ES model that
> >> they are using to calculate required regulatory capital.
> >
> > In my opinion, there is one aspect that introduces some confusion. ES
> > (CVaR) is now common, but many people, perhaps out of habit, maybe for
> > historical reasons, still use the term VaR instead of the correct name
> > (ES).
>
> Not sure I follow. VaR and ES are different measures. VaR is a
> quantile while ES is the average loss conditional on that quantile
> (i.e. the expected loss conditional that the loss is greater than
> the quantile of the loss distribution).

I agree that these names should not be confused. However, I
encountered that the _name_ "VaR" is used for ES. In my opinion, this
is due to a mental shortcut, or it's a historical habit. Such
imprecise use of the names often leads to misunderstanding.

Daniel

> Regards,
>
> Alexios
>
> >
> > Best regards,
> > Daniel
> >
> >
> >> Regards,
> >>
> >> Brian
> >>
> >>
> >> --
> >> Brian G. Peterson
> >> ph: +1.773.459.4973
> >> im: bgpbraverock
> >>
> >> _______________________________________________
> >> [hidden email] mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> >> -- Subscriber-posting only. If you want to post, subscribe first.
> >> -- Also note that this is not the r-help list where general R questions should go.
> >
> > _______________________________________________
> > [hidden email] mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> > -- Subscriber-posting only. If you want to post, subscribe first.
> > -- Also note that this is not the r-help list where general R questions should go.
> >

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Back testing

Bogaso
In reply to this post by leo sea
Thanks Brian, the resources are really helpful.

However I am not sure if I fully understood the implementation part of
the MSCI's approach. It basically defines different test-statistics
r.g. Z1, Z2, etc. For Z1, it asserts that under null, the expected
value for Z1 will be zero. I failed to see what distribution would it
take under H0, so that I can complete the significance testing and/or
defining some confidence interval under null.

Ideally, with realised daily PnL and forecasted ES, we will have a
time series of Z1 - if my understanding is perfect. To carry out if
E[Z1] = 0, can I do some t-test or some non-parametric test for
testing mean =0?

I think, this should be valid as only assumption was that PnL has to
be independent, may not be identically distributed. My only concern
is, can I use an ordinary significance table for t-test? I am little
concerned because, testing would be done on Z1's values, which are
calculated values, not the original dataset. So a non-parametric test
may be more appropriate.

Any pointer on above thinking is highly appreciated.

On Wed, Jun 10, 2020 at 5:21 PM leo sea <[hidden email]> wrote:

>
> Hi
> I am also interested in in ES backtesting.
> Good idea
> Thanks
>
> Téléchargez Outlook pour Android
>
> ________________________________
> From: R-SIG-Finance <[hidden email]> on behalf of Christofer Bogaso <[hidden email]>
> Sent: Wednesday, June 10, 2020 11:38:57 AM
> To: [hidden email] <[hidden email]>
> Subject: [R-SIG-Finance] Back testing
>
> Hi,
>
> I was looking for an idea how banks backtest their models for Expected
> Shortfall. Backtesting VaR is well documented but I failed to get any
> practical idea about backtesting ES.
>
> Any pointer towards the best practice will be really helpful.
>
> Thanks,
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions should go.

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Back testing Expected Shortfall

Pit Götz
In reply to this post by Daniel Cegiełka
Hello everyone,

I work at a university in germany and we are also currently working on forecasting ES and (of course) backtesting of said forecasts.

Over the last few months some students, who are writing their masters thesis at our chair, had to some litarature research.
Thats why I wanted to give you a very brief overview of their findings:

The most widely applied ES backtests seems to be the backtest by McNeil, Frey and Embrechts (2000), implemented for example in the rugarch package.
(the test was already mentioned here by Alexios)

In addition to the already mentioned tests and the paper by Acerby and Szekely I wanted to add the following:

A Hitsequence based backtest was introduced for by Du, Escanciano (2017). As far as I am concerned, this test has not yet been implemented in a package, but their code is available online. In a broader view, this test is a special case of a spectral measure test by Costanzino, Curran (2014), which was then extended to a Basel-Like traffic light approach in 2018 (Not sure about the availability of code).

In Emmer et al. (2015) it is suggested, that a suitable ES forecast can be approximated by only 4 different VaR forecasts. This also suggests, that you can backtest ES, forecasted by a model that forecasts both, ES and VaR, such as GARCH, by backtesting th 4 different VaR forecasts.
However this approach seems to need more empirical valuation.

I also wanted to mention the paper by Gneiting (2011), showing that the ES lacks elicitability property. This can lead to complications, when you try to backtest the ES itself as a point forecast.However, this property can be used to construct a model comparison like backtest as in Fissler et al. (2015).

More reacently, a quantile regression based approach has been suggested by Coupier, Leymarie (2020). I have not yet read said paper and therefore I can not tell you anything about it.

I hope that this message gives you some new insights and some usefull information.

Best regards,
Pit
 


Research Associate

Martin-Luther-Universität Halle-Wittenberg

Chair of Finance & Banking

 

Große Steinstraße 73 | D-06108 Halle | Germany
Tel 0049 345 5523452


>>> Daniel Cegiełka <[hidden email]> 10.06.20 21.49 Uhr >>>
śr., 10 cze 2020 o 21:14 alexios galanos <[hidden email]> napisał(a):

>
>
>
> On 6/10/20 11:08 AM, Daniel Cegiełka wrote:
> > śr., 10 cze 2020 o 19:23 Brian G. Peterson <[hidden email]> napisał(a):
> >>
> >> On Wed, 2020-06-10 at 15:08 +0530, Christofer Bogaso wrote:
> >>> I was looking for an idea how banks backtest their models for
> >>> Expected
> >>> Shortfall. Backtesting VaR is well documented but I failed to get any
> >>> practical idea about backtesting ES.
> >>>
> >>> Any pointer towards the best practice will be really helpful.
> >>
> >> If you are using Normal VaR, then you know the Expected Shortfall
> >> estimate too.
> >>
> >> If you are using a different mechanism, then of course the mean loss
> >> when the loss exceeds the VaR may be significantly different than the
> >> Normal ES.
> >>
> >> So, to backetesting... the newest Basel standard replaces VaR with ES,
> >> and requires that banks justify their use of a particular ES model that
> >> they are using to calculate required regulatory capital.
> >
> > In my opinion, there is one aspect that introduces some confusion. ES
> > (CVaR) is now common, but many people, perhaps out of habit, maybe for
> > historical reasons, still use the term VaR instead of the correct name
> > (ES).
>
> Not sure I follow. VaR and ES are different measures. VaR is a
> quantile while ES is the average loss conditional on that quantile
> (i.e. the expected loss conditional that the loss is greater than
> the quantile of the loss distribution).

I agree that these names should not be confused. However, I
encountered that the _name_ "VaR" is used for ES. In my opinion, this
is due to a mental shortcut, or it's a historical habit. Such
imprecise use of the names often leads to misunderstanding.

Daniel

> Regards,
>
> Alexios
>
> >
> > Best regards,
> > Daniel
> >
> >
> >> Regards,
> >>
> >> Brian
> >>
> >>
> >> --
> >> Brian G. Peterson
> >> ph: +1.773.459.4973
> >> im: bgpbraverock
> >>
> >> _______________________________________________
> >> [hidden email] mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> >> -- Subscriber-posting only. If you want to post, subscribe first.
> >> -- Also note that this is not the r-help list where general R questions should go.
> >
> > _______________________________________________
> > [hidden email] mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> > -- Subscriber-posting only. If you want to post, subscribe first.
> > -- Also note that this is not the r-help list where general R questions should go.
> >

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Back testing Expected Shortfall

alexios
Hi Pit and thanks for sharing.

I was not aware of the Gneiting paper, but the Gneiting and Raftery
(2007) paper discusses scoring rules and their mean interval score (MIS)
has been used in the M4 competition (implemented in the greybox package).

Best,

Alexios

On 6/15/20 7:34 AM, Pit Götz wrote:

> Hello everyone,
>
> I work at a university in germany and we are also currently working on
> forecasting ES and (of course) backtesting of said forecasts.
>
> Over the last few months some students, who are writing their masters
> thesis at our chair, had to some litarature research.
> Thats why I wanted to give you a very brief overview of their findings:
>
> The most widely applied ES backtests seems to be the backtest by McNeil,
> Frey and Embrechts (2000), implemented for example in the rugarch package.
> (the test was already mentioned here by Alexios)
>
> In addition to the already mentioned tests and the paper by Acerby and
> Szekely I wanted to add the following:
>
> A Hitsequence based backtest was introduced for by Du, Escanciano
> (2017). As far as I am concerned, this test has not yet been implemented
> in a package, but their code is available online. In a broader view,
> this test is a special case of a spectral measure test by Costanzino,
> Curran (2014), which was then extended to a Basel-Like traffic light
> approach in 2018 (Not sure about the availability of code).
>
> In Emmer et al. (2015) it is suggested, that a suitable ES forecast can
> be approximated by only 4 different VaR forecasts. This also suggests,
> that you can backtest ES, forecasted by a model that forecasts both, ES
> and VaR, such as GARCH, by backtesting th 4 different VaR forecasts.
> However this approach seems to need more empirical valuation.
>
> I also wanted to mention the paper by Gneiting (2011), showing that the
> ES lacks elicitability property. This can lead to complications, when
> you try to backtest the ES itself as a point forecast.However, this
> property can be used to construct a model comparison like backtest as in
> Fissler et al. (2015).
>
> More reacently, a quantile regression based approach has been suggested
> by Coupier, Leymarie (2020). I have not yet read said paper and
> therefore I can not tell you anything about it.
>
> I hope that this message gives you some new insights and some usefull
> information.
>
> Best regards,
> Pit
>
>
>
> Research Associate
>
> *Martin-Luther-Universität Halle-Wittenberg*
>
> Chair of Finance & Banking
>
> Große Steinstraße 73 | D-06108 Halle | Germany
> Tel 0049 345 5523452
>
>
>>>> Daniel Cegiełka <[hidden email]> 10.06.20 21.49 Uhr >>>
> śr., 10 cze 2020 o 21:14 alexios galanos <[hidden email]> napisał(a):
>  >
>  >
>  >
>  > On 6/10/20 11:08 AM, Daniel Cegiełka wrote:
>  > > śr., 10 cze 2020 o 19:23 Brian G. Peterson <[hidden email]>
> napisał(a):
>  > >>
>  > >> On Wed, 2020-06-10 at 15:08 +0530, Christofer Bogaso wrote:
>  > >>> I was looking for an idea how banks backtest their models for
>  > >>> Expected
>  > >>> Shortfall. Backtesting VaR is well documented but I failed to get any
>  > >>> practical idea about backtesting ES.
>  > >>>
>  > >>> Any pointer towards the best practice will be really helpful.
>  > >>
>  > >> If you are using Normal VaR, then you know the Expected Shortfall
>  > >> estimate too.
>  > >>
>  > >> If you are using a different mechanism, then of course the mean loss
>  > >> when the loss exceeds the VaR may be significantly different than the
>  > >> Normal ES.
>  > >>
>  > >> So, to backetesting... the newest Basel standard replaces VaR with ES,
>  > >> and requires that banks justify their use of a particular ES model
> that
>  > >> they are using to calculate required regulatory capital.
>  > >
>  > > In my opinion, there is one aspect that introduces some confusion. ES
>  > > (CVaR) is now common, but many people, perhaps out of habit, maybe for
>  > > historical reasons, still use the term VaR instead of the correct name
>  > > (ES).
>  >
>  > Not sure I follow. VaR and ES are different measures. VaR is a
>  > quantile while ES is the average loss conditional on that quantile
>  > (i.e. the expected loss conditional that the loss is greater than
>  > the quantile of the loss distribution).
>
> I agree that these names should not be confused. However, I
> encountered that the _name_ "VaR" is used for ES. In my opinion, this
> is due to a mental shortcut, or it's a historical habit. Such
> imprecise use of the names often leads to misunderstanding.
>
> Daniel
>
>  > Regards,
>  >
>  > Alexios
>  >
>  > >
>  > > Best regards,
>  > > Daniel
>  > >
>  > >
>  > >> Regards,
>  > >>
>  > >> Brian
>  > >>
>  > >>
>  > >> --
>  > >> Brian G. Peterson
>  > >> ph: +1.773.459.4973
>  > >> im: bgpbraverock
>  > >>
>  > >> _______________________________________________
>  > >> [hidden email] mailing list
>  > >> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>  > >> -- Subscriber-posting only. If you want to post, subscribe first.
>  > >> -- Also note that this is not the r-help list where general R
> questions should go.
>  > >
>  > > _______________________________________________
>  > > [hidden email] mailing list
>  > > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>  > > -- Subscriber-posting only. If you want to post, subscribe first.
>  > > -- Also note that this is not the r-help list where general R
> questions should go.
>  > >
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions
> should go.

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Back testing Expected Shortfall

Sebastian Bayer
Hi all,

We have been working on an ES backtest that only requires expected
shortfall forecasts (no quantiles or other inputs) besides the returns.
This is in striking contrast to all other available backtests.

The paper is going to be published in the next few days in JFEC. This is
the abstract:

This paper introduces novel backtests for the risk measure Expected
Shortfall (ES) following the testing idea
of Mincer and Zarnowitz (1969). Estimating a regression model for the ES
stand-alone is infeasible, and thus,
our tests are based on a joint regression model for the Value at Risk and
the ES, which allows for different
test specifications. These ES backtests are the first which solely backtest
the ES in the sense that they only
require ES forecasts as input variables. As the tests are potentially
subject to model misspecification, we
provide asymptotic theory under misspecification for the underlying joint
regression. We find that employing
a misspecification robust covariance estimator substantially improves the
tests’ performance. We compare our
backtests to existing joint VaR and ES backtests and find that our tests
outperform the existing alternatives
throughout all considered simulations. In an empirical illustration, we
apply our backtests to ES forecasts for
200 stocks of the S&P 500 index.

You can find the last working paper version here:
https://arxiv.org/pdf/1801.04112.pdf
We also have an R package:
https://cran.r-project.org/web/packages/esback/index.html
Here's an example how to apply the backtest:
https://github.com/BayerSe/esback#examples

Regards

Am Mo., 15. Juni 2020 um 17:01 Uhr schrieb alexios galanos <
[hidden email]>:

> Hi Pit and thanks for sharing.
>
> I was not aware of the Gneiting paper, but the Gneiting and Raftery
> (2007) paper discusses scoring rules and their mean interval score (MIS)
> has been used in the M4 competition (implemented in the greybox package).
>
> Best,
>
> Alexios
>
> On 6/15/20 7:34 AM, Pit Götz wrote:
> > Hello everyone,
> >
> > I work at a university in germany and we are also currently working on
> > forecasting ES and (of course) backtesting of said forecasts.
> >
> > Over the last few months some students, who are writing their masters
> > thesis at our chair, had to some litarature research.
> > Thats why I wanted to give you a very brief overview of their findings:
> >
> > The most widely applied ES backtests seems to be the backtest by McNeil,
> > Frey and Embrechts (2000), implemented for example in the rugarch
> package.
> > (the test was already mentioned here by Alexios)
> >
> > In addition to the already mentioned tests and the paper by Acerby and
> > Szekely I wanted to add the following:
> >
> > A Hitsequence based backtest was introduced for by Du, Escanciano
> > (2017). As far as I am concerned, this test has not yet been implemented
> > in a package, but their code is available online. In a broader view,
> > this test is a special case of a spectral measure test by Costanzino,
> > Curran (2014), which was then extended to a Basel-Like traffic light
> > approach in 2018 (Not sure about the availability of code).
> >
> > In Emmer et al. (2015) it is suggested, that a suitable ES forecast can
> > be approximated by only 4 different VaR forecasts. This also suggests,
> > that you can backtest ES, forecasted by a model that forecasts both, ES
> > and VaR, such as GARCH, by backtesting th 4 different VaR forecasts.
> > However this approach seems to need more empirical valuation.
> >
> > I also wanted to mention the paper by Gneiting (2011), showing that the
> > ES lacks elicitability property. This can lead to complications, when
> > you try to backtest the ES itself as a point forecast.However, this
> > property can be used to construct a model comparison like backtest as in
> > Fissler et al. (2015).
> >
> > More reacently, a quantile regression based approach has been suggested
> > by Coupier, Leymarie (2020). I have not yet read said paper and
> > therefore I can not tell you anything about it.
> >
> > I hope that this message gives you some new insights and some usefull
> > information.
> >
> > Best regards,
> > Pit
> >
> >
> >
> > Research Associate
> >
> > *Martin-Luther-Universität Halle-Wittenberg*
> >
> > Chair of Finance & Banking
> >
> > Große Steinstraße 73 | D-06108 Halle | Germany
> > Tel 0049 345 5523452
> >
> >
> >>>> Daniel Cegiełka <[hidden email]> 10.06.20 21.49 Uhr >>>
> > śr., 10 cze 2020 o 21:14 alexios galanos <[hidden email]>
> napisał(a):
> >  >
> >  >
> >  >
> >  > On 6/10/20 11:08 AM, Daniel Cegiełka wrote:
> >  > > śr., 10 cze 2020 o 19:23 Brian G. Peterson <[hidden email]>
> > napisał(a):
> >  > >>
> >  > >> On Wed, 2020-06-10 at 15:08 +0530, Christofer Bogaso wrote:
> >  > >>> I was looking for an idea how banks backtest their models for
> >  > >>> Expected
> >  > >>> Shortfall. Backtesting VaR is well documented but I failed to get
> any
> >  > >>> practical idea about backtesting ES.
> >  > >>>
> >  > >>> Any pointer towards the best practice will be really helpful.
> >  > >>
> >  > >> If you are using Normal VaR, then you know the Expected Shortfall
> >  > >> estimate too.
> >  > >>
> >  > >> If you are using a different mechanism, then of course the mean
> loss
> >  > >> when the loss exceeds the VaR may be significantly different than
> the
> >  > >> Normal ES.
> >  > >>
> >  > >> So, to backetesting... the newest Basel standard replaces VaR with
> ES,
> >  > >> and requires that banks justify their use of a particular ES model
> > that
> >  > >> they are using to calculate required regulatory capital.
> >  > >
> >  > > In my opinion, there is one aspect that introduces some confusion.
> ES
> >  > > (CVaR) is now common, but many people, perhaps out of habit, maybe
> for
> >  > > historical reasons, still use the term VaR instead of the correct
> name
> >  > > (ES).
> >  >
> >  > Not sure I follow. VaR and ES are different measures. VaR is a
> >  > quantile while ES is the average loss conditional on that quantile
> >  > (i.e. the expected loss conditional that the loss is greater than
> >  > the quantile of the loss distribution).
> >
> > I agree that these names should not be confused. However, I
> > encountered that the _name_ "VaR" is used for ES. In my opinion, this
> > is due to a mental shortcut, or it's a historical habit. Such
> > imprecise use of the names often leads to misunderstanding.
> >
> > Daniel
> >
> >  > Regards,
> >  >
> >  > Alexios
> >  >
> >  > >
> >  > > Best regards,
> >  > > Daniel
> >  > >
> >  > >
> >  > >> Regards,
> >  > >>
> >  > >> Brian
> >  > >>
> >  > >>
> >  > >> --
> >  > >> Brian G. Peterson
> >  > >> ph: +1.773.459.4973
> >  > >> im: bgpbraverock
> >  > >>
> >  > >> _______________________________________________
> >  > >> [hidden email] mailing list
> >  > >> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> >  > >> -- Subscriber-posting only. If you want to post, subscribe first.
> >  > >> -- Also note that this is not the r-help list where general R
> > questions should go.
> >  > >
> >  > > _______________________________________________
> >  > > [hidden email] mailing list
> >  > > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> >  > > -- Subscriber-posting only. If you want to post, subscribe first.
> >  > > -- Also note that this is not the r-help list where general R
> > questions should go.
> >  > >
> >
> > _______________________________________________
> > [hidden email] mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> > -- Subscriber-posting only. If you want to post, subscribe first.
> > -- Also note that this is not the r-help list where general R questions
> > should go.
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions
> should go.
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.