Can anybody help me understand AIC and BIC and devise a new metric?

[LosemindL]

Hi all,

Could anybody please help me understand AIC and BIC and especially why do they make sense?

Furthermore, I am trying to devise a new metric related to the model selection in the financial asset management industry.

As you know the industry uses Sharpe Ratio as the main performance benchmark, which is the annualized mean of returns divided by the annualized standard deviation of returns.

In model selection, we would like to choose a model that yields the highest Sharpe Ratio.

However, the more parameters you use, the higher Sharpe Ratio you might potentially get, and the higher risk that your model is overfitted.

I am trying to think of a AIC or BIC version of the Sharpe Ratio that facilitates the model selection...

Anybody could you please give me some pointers?

Thanks a lot!

[David Winsemius]

On Jul 5, 2010, at 10:35 AM, LosemindL wrote:

>
> Hi all,
>
> Could anybody please help me understand AIC and BIC and especially  
> why do
> they make sense?
>
> Furthermore, I am trying to devise a new metric related to the model
> selection in the financial asset management industry.
>
> As you know the industry uses Sharpe Ratio as the main performance
> benchmark, which is the annualized mean of returns divided by the  
> annualized
> standard deviation of returns.
>
> In model selection, we would like to choose a model that yields the  
> highest
> Sharpe Ratio.
>
> However, the more parameters you use, the higher Sharpe Ratio you  
> might
> potentially get, and the higher risk that your model is overfitted.
>
> I am trying to think of a AIC or BIC version of the Sharpe Ratio that
> facilitates the model selection...
>
> Anybody could you please give me some pointers?
>

From: http://www.R-project.org/posting-guide.html

"Basic statistics and classroom homework: R-help is not intended for  
these."

Perhaps following some link on Wikipedia, instead?

--
David Winsemius, MD
West Hartford, CT

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[djmuseR]

Hi:

On Mon, Jul 5, 2010 at 7:35 AM, LosemindL <[hidden email]> wrote:

>
> Hi all,
>
> Could anybody please help me understand AIC and BIC and especially why do
> they make sense?
>

Any good text that discusses model selection in detail will have some
discussion of
AIC and BIC. Frank Harrell's book 'Regression Modeling Strategies' comes
immediately
to mind, along with Hastie, Tibshirani and Friedman (Elements of Statistical
Learning)
and Burnham and Anderson's book (Model Selection and Multi-Model Inference),
but
there are many other worthy texts that cover the topic. The gist is that AIC
and BIC
penalize the log likelihood of a model by subtracting different functions of
its number
of parameters. David's suggestion of Wikipedia is also on target.

>
> Furthermore, I am trying to devise a new metric related to the model
> selection in the financial asset management industry.
>
> As you know the industry uses Sharpe Ratio as the main performance
> benchmark, which is the annualized mean of returns divided by the
> annualized
> standard deviation of returns.
>

I didn't know, but thank you for the information. Isn't this simply a
signal-to-noise
ratio quantified on an annual basis?

>
> In model selection, we would like to choose a model that yields the highest
> Sharpe Ratio.
>
> However, the more parameters you use, the higher Sharpe Ratio you might
> potentially get, and the higher risk that your model is overfitted.
>
> I am trying to think of a AIC or BIC version of the Sharpe Ratio that
> facilitates the model selection...
>

You might be able to make some progress if you can express the (penalized)
log likelihood as a function of the Sharpe ratio. But if you have several
years of
data in your model and the ratio is computed annually, then isn't it a
random
variable rather than a parameter? If so, it changes the nature of the
problem, no?
(Being unfamiliar with the Sharpe ratio, I fully recognize that I may be
completely
off-base in this suggestion, but I'll put it out there anyway :)

BTW, you might find the R-sig-finance list to be a more productive resource
in
this problem than R-help due to the specialized nature of the question.

HTH,
Dennis

>
> Anybody could you please give me some pointers?
>
> Thanks a lot!
> --
> View this message in context:
> http://r.789695.n4.nabble.com/Can-anybody-help-me-understand-AIC-and-BIC-and-devise-a-new-metric-tp2278448p2278448.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[Kjetil Halvorsen]

You should have a look at:

"Model Selection and
Model Averaging"
Gerda Claeskens
K.U. Leuven
Nils Lid Hjort
University of Oslo

Among other this will explain that AIC and BIC really aims at different goals.

On Mon, Jul 5, 2010 at 4:20 PM, Dennis Murphy <[hidden email]> wrote:

> Hi:
>
> On Mon, Jul 5, 2010 at 7:35 AM, LosemindL <[hidden email]> wrote:
>
>>
>> Hi all,
>>
>> Could anybody please help me understand AIC and BIC and especially why do
>> they make sense?
>>
>
> Any good text that discusses model selection in detail will have some
> discussion of
> AIC and BIC. Frank Harrell's book 'Regression Modeling Strategies' comes
> immediately
> to mind, along with Hastie, Tibshirani and Friedman (Elements of Statistical
> Learning)
> and Burnham and Anderson's book (Model Selection and Multi-Model Inference),
> but
> there are many other worthy texts that cover the topic. The gist is that AIC
> and BIC
> penalize the log likelihood of a model by subtracting different functions of
> its number
> of parameters. David's suggestion of Wikipedia is also on target.
>
>>
>> Furthermore, I am trying to devise a new metric related to the model
>> selection in the financial asset management industry.
>>
>> As you know the industry uses Sharpe Ratio as the main performance
>> benchmark, which is the annualized mean of returns divided by the
>> annualized
>> standard deviation of returns.
>>
>
> I didn't know, but thank you for the information. Isn't this simply a
> signal-to-noise
> ratio quantified on an annual basis?
>
>>
>> In model selection, we would like to choose a model that yields the highest
>> Sharpe Ratio.
>>
>> However, the more parameters you use, the higher Sharpe Ratio you might
>> potentially get, and the higher risk that your model is overfitted.
>>
>> I am trying to think of a AIC or BIC version of the Sharpe Ratio that
>> facilitates the model selection...
>>
>
> You might be able to make some progress if you can express the (penalized)
> log likelihood as a function of the Sharpe ratio. But if you have several
> years of
> data in your model and the ratio is computed annually, then isn't it a
> random
> variable rather than a parameter? If so, it changes the nature of the
> problem, no?
> (Being unfamiliar with the Sharpe ratio, I fully recognize that I may be
> completely
> off-base in this suggestion, but I'll put it out there anyway :)
>
> BTW, you might find the R-sig-finance list to be a more productive resource
> in
> this problem than R-help due to the specialized nature of the question.
>
> HTH,
> Dennis
>
>>
>> Anybody could you please give me some pointers?
>>
>> Thanks a lot!
>> --
>> View this message in context:
>> http://r.789695.n4.nabble.com/Can-anybody-help-me-understand-AIC-and-BIC-and-devise-a-new-metric-tp2278448p2278448.html
>> Sent from the R help mailing list archive at Nabble.com.
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
>        [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[Stephan Kolassa]

Hi,

one comment: Claeskens and Hjort define AIC as 2*log L - 2*p for a model
with likelihood L and p parameters; consequently, they look for models
with *maximum* AIC in model selection and averaging. This differs from
the vast majority of authors (and R), who define AIC as -2*log L + 2*p
and search for the model with *minimum* AIC. Their definition of BIC is
similarly the negative of "normal" BIC.

I would compare this to defining \pi as the base of the natural
logarithm and e as the ratio of a circle's circumference to its
diameter: of course, you can do perfectly valid mathematics with your
own definitions, but it is a recipe for confusion.

Anyone who only reads Claeskens and Hjort, fires up R and selects the
model with the maximum AIC from the candidate models is in for some
*nasty* surprises.

Worse, as far as I see, Claeskens and Hjort nowhere mention that they
are using a definition that is diametrically opposed to what is
(overwhelmingly) common, and they do not comment on this.

However, Claeskens and Hjort managed to publish a book, which I have yet
to do, so it is quite possible that there is a major flaw in my
thinking. If so, I haven't found it yet, and I would be very grateful if
somebody pointed out what I misunderstand.

Otherwise, I would be *very* careful indeed about basing my analysis
strategy on their book, although the rest of the content is very helpful
indeed - you only need to remember where to switch signs and change
"maximize" to "minimize" etc.

For AIC and BIC novices, I would recommend going with Burnham &
Anderson, which Kjetil cited below.

Best,
Stephan



Kjetil Halvorsen schrieb:

> You should have a look at:
>
> "Model Selection and
> Model Averaging"
> Gerda Claeskens
> K.U. Leuven
> Nils Lid Hjort
> University of Oslo
>
> Among other this will explain that AIC and BIC really aims at different goals.
>
> On Mon, Jul 5, 2010 at 4:20 PM, Dennis Murphy <[hidden email]> wrote:
>> Hi:
>>
>> On Mon, Jul 5, 2010 at 7:35 AM, LosemindL <[hidden email]> wrote:
>>
>>> Hi all,
>>>
>>> Could anybody please help me understand AIC and BIC and especially why do
>>> they make sense?
>>>
>> Any good text that discusses model selection in detail will have some
>> discussion of
>> AIC and BIC. Frank Harrell's book 'Regression Modeling Strategies' comes
>> immediately
>> to mind, along with Hastie, Tibshirani and Friedman (Elements of Statistical
>> Learning)
>> and Burnham and Anderson's book (Model Selection and Multi-Model Inference),
>> but
>> there are many other worthy texts that cover the topic. The gist is that AIC
>> and BIC
>> penalize the log likelihood of a model by subtracting different functions of
>> its number
>> of parameters. David's suggestion of Wikipedia is also on target.
>>
>>> Furthermore, I am trying to devise a new metric related to the model
>>> selection in the financial asset management industry.
>>>
>>> As you know the industry uses Sharpe Ratio as the main performance
>>> benchmark, which is the annualized mean of returns divided by the
>>> annualized
>>> standard deviation of returns.
>>>
>> I didn't know, but thank you for the information. Isn't this simply a
>> signal-to-noise
>> ratio quantified on an annual basis?
>>
>>> In model selection, we would like to choose a model that yields the highest
>>> Sharpe Ratio.
>>>
>>> However, the more parameters you use, the higher Sharpe Ratio you might
>>> potentially get, and the higher risk that your model is overfitted.
>>>
>>> I am trying to think of a AIC or BIC version of the Sharpe Ratio that
>>> facilitates the model selection...
>>>
>> You might be able to make some progress if you can express the (penalized)
>> log likelihood as a function of the Sharpe ratio. But if you have several
>> years of
>> data in your model and the ratio is computed annually, then isn't it a
>> random
>> variable rather than a parameter? If so, it changes the nature of the
>> problem, no?
>> (Being unfamiliar with the Sharpe ratio, I fully recognize that I may be
>> completely
>> off-base in this suggestion, but I'll put it out there anyway :)
>>
>> BTW, you might find the R-sig-finance list to be a more productive resource
>> in
>> this problem than R-help due to the specialized nature of the question.
>>
>> HTH,
>> Dennis
>>
>>> Anybody could you please give me some pointers?
>>>
>>> Thanks a lot!
>>> --
>>> View this message in context:
>>> http://r.789695.n4.nabble.com/Can-anybody-help-me-understand-AIC-and-BIC-and-devise-a-new-metric-tp2278448p2278448.html
>>> Sent from the R help mailing list archive at Nabble.com.
>>>
>>> ______________________________________________
>>> [hidden email] mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>        [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.