Failure of solve.QP in portfolio modeling

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Failure of solve.QP in portfolio modeling

Alec Schmidt-2
Hi everyone,
I'm trying to estimate optimal Markowitz portfolio weights for a list of
stocks chosen upon some criterion using solve.QP from quadprog library.
When the number of stocks N reaches some limit, I get a message "matrix
D in quadratic function is not positive definite." For example, if I
rebalance every 6 weeks (which implies that variance is calculated for
6-week interval prior to the period for which I calculate portfolio
weights), I can get solution for 25>=N<50. For 12-week interval,
solution exists for 50>=N<100, and for 24-week interval, I can get
solution for N=100. My attempt to remedy this problem with Higham's
method doesn't help. I'll greatly appreciate you input: first, why this
may happen (can there be lack of local minimum?), and second, whether
there are R solvers that may need only semi positive definite matrix.

Thanks! Alec

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Failure of solve.QP in portfolio modeling

Ilya Kipnis
Because you can't invert a matrix with more features than observations. IE
if you have 50 stocks but only use 26 weekly return data points, you can't
invert that covariance matrix because it's not positive semi-definite. You
may want to try my Constrained Critical Line algorithm.

https://quantstrattrader.wordpress.com/2015/06/05/momentum-markowitz-and-solving-rank-deficient-covariance-matrices-the-constrained-critical-line-algorithm/



On Tue, Sep 22, 2015 at 9:37 AM, aschmid1 <[hidden email]> wrote:

> Hi everyone,
> I'm trying to estimate optimal Markowitz portfolio weights for a list of
> stocks chosen upon some criterion using solve.QP from quadprog library.
> When the number of stocks N reaches some limit, I get a message "matrix D
> in quadratic function is not positive definite." For example, if I
> rebalance every 6 weeks (which implies that variance is calculated for
> 6-week interval prior to the period for which I calculate portfolio
> weights), I can get solution for 25>=N<50. For 12-week interval, solution
> exists for 50>=N<100, and for 24-week interval, I can get solution for
> N=100. My attempt to remedy this problem with Higham's method doesn't help.
> I'll greatly appreciate you input: first, why this may happen (can there be
> lack of local minimum?), and second, whether there are R solvers that may
> need only semi positive definite matrix.
>
> Thanks! Alec
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions
> should go.
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Failure of solve.QP in portfolio modeling

Enrico Schumann-2
In reply to this post by Alec Schmidt-2
On Tue, 22 Sep 2015, aschmid1 <[hidden email]> writes:

> Hi everyone,
> I'm trying to estimate optimal Markowitz portfolio weights for a list
> of stocks chosen upon some criterion using solve.QP from quadprog
> library. When the number of stocks N reaches some limit, I get a
> message "matrix D in quadratic function is not positive definite." For
> example, if I rebalance every 6 weeks (which implies that variance is
> calculated for 6-week interval prior to the period for which I
> calculate portfolio weights), I can get solution for 25>=N<50. For
> 12-week interval, solution exists for 50>=N<100, and for 24-week
> interval, I can get solution for N=100. My attempt to remedy this
> problem with Higham's method doesn't help. I'll greatly appreciate you
> input: first, why this may happen (can there be lack of local
> minimum?), and second, whether there are R solvers that may need only
> semi positive definite matrix.
>
> Thanks! Alec
>

The thing you may want to look up is the "rank" of a matrix.

For instance, I create a small data set R -- suppose these were
daily-returns data of 10 equities.

  na   <- 10 ## number of assets
  nobs <- 10 ## number of observations

  R <- array(rnorm(nobs * na, sd = 0.01), dim = c(nobs, na))
  qr(cov(R))$rank

The rank of the covariance matrix is only 9; you need na+1 observations
to get full rank. You can still compute the standard deviation of a
portfolio:

  ew <- rep(1/na, na) ## equal-weight portfolio
  sqrt(ew %*% cov(R) %*% ew)

But with a non-full rank matrix and no constraints, it is guaranteed
that you have portfolios like this one:

  zerovol <- svd(cov(R))$v[,10]
  sqrt(abs(zerovol %*% cov(R) %*% zerovol))

You get a zero-volatility portfolio.

Whether that matters depends on your application. With constraints,
perhaps not.

An example in which it does not matter is in Section "1.3
Redundant assets" in
https://cran.r-project.org/web/packages/NMOF/vignettes/TAportfolio.pdf 

There you also have an example of a solver. A lengthier discussion is in
Section "13.2.5 Repairing Matrices" in this book

@BOOK{Gilli2011b,
  title        = {Numerical Methods and Optimization in Finance},
  publisher    = {Elsevier/Academic Press},
  year         = 2011,
  author       = {Gilli, Manfred and Maringer, Dietmar and Schumann,
                  Enrico}
}

of which [disclosure], I am a co-author.

--
Enrico Schumann
Lucerne, Switzerland
http://enricoschumann.net

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Failure of solve.QP in portfolio modeling

Patrick Burns-2
In reply to this post by Alec Schmidt-2
You can use a factor model or shrinkage
to get a positive definite variance matrix.
There is a function for each in the
BurStFin package on CRAN.

The optimizer in Portfolio Probe doesn't
care about positive definiteness (though
that is not always a good thing).  It is
free for academic use.

Pat

On 22/09/2015 14:37, aschmid1 wrote:

> Hi everyone,
> I'm trying to estimate optimal Markowitz portfolio weights for a list of
> stocks chosen upon some criterion using solve.QP from quadprog library.
> When the number of stocks N reaches some limit, I get a message "matrix
> D in quadratic function is not positive definite." For example, if I
> rebalance every 6 weeks (which implies that variance is calculated for
> 6-week interval prior to the period for which I calculate portfolio
> weights), I can get solution for 25>=N<50. For 12-week interval,
> solution exists for 50>=N<100, and for 24-week interval, I can get
> solution for N=100. My attempt to remedy this problem with Higham's
> method doesn't help. I'll greatly appreciate you input: first, why this
> may happen (can there be lack of local minimum?), and second, whether
> there are R solvers that may need only semi positive definite matrix.
>
> Thanks! Alec
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions
> should go.

--
Patrick Burns
[hidden email]
http://www.burns-stat.com
http://www.portfolioprobe.com/blog
twitter: @burnsstat @portfolioprobe

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Failure of solve.QP in portfolio modeling

Patrick Burns-2
One correction and another comment.

I should have said that a trade optimizer
not caring about positive definiteness is
usually a bad thing -- it has its uses
but giving the optimizer a shot at a
portfolio with negative variance is asking
for trouble (because it will do its best
to get there).

The estimates of the variance with only a
few observations are going to be extremely
noisy.  I think the length of history to
be used to estimate the variance should
depend on what length gives the best variance
estimate and should be independent of the
rebalancing schedule.  (Though perhaps I
have misinterpreted the original post.)

Pat

On 22/09/2015 19:59, Patrick Burns wrote:

> You can use a factor model or shrinkage
> to get a positive definite variance matrix.
> There is a function for each in the
> BurStFin package on CRAN.
>
> The optimizer in Portfolio Probe doesn't
> care about positive definiteness (though
> that is not always a good thing).  It is
> free for academic use.
>
> Pat
>
> On 22/09/2015 14:37, aschmid1 wrote:
>> Hi everyone,
>> I'm trying to estimate optimal Markowitz portfolio weights for a list of
>> stocks chosen upon some criterion using solve.QP from quadprog library.
>> When the number of stocks N reaches some limit, I get a message "matrix
>> D in quadratic function is not positive definite." For example, if I
>> rebalance every 6 weeks (which implies that variance is calculated for
>> 6-week interval prior to the period for which I calculate portfolio
>> weights), I can get solution for 25>=N<50. For 12-week interval,
>> solution exists for 50>=N<100, and for 24-week interval, I can get
>> solution for N=100. My attempt to remedy this problem with Higham's
>> method doesn't help. I'll greatly appreciate you input: first, why this
>> may happen (can there be lack of local minimum?), and second, whether
>> there are R solvers that may need only semi positive definite matrix.
>>
>> Thanks! Alec
>>
>> _______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
>> -- Subscriber-posting only. If you want to post, subscribe first.
>> -- Also note that this is not the r-help list where general R questions
>> should go.
>

--
Patrick Burns
[hidden email]
http://www.burns-stat.com
http://www.portfolioprobe.com/blog
twitter: @burnsstat @portfolioprobe

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.
Reply | Threaded
Open this post in threaded view
|

Re: Failure of solve.QP in portfolio modeling

Alexandre Shannon
Hello Alec

Package "corpcor" has a useful function called "make.positive.definite",
which might help in your particular case.

Alex
On Tue, Sep 22, 2015 at 16:03 Patrick Burns <[hidden email]> wrote:

> One correction and another comment.
>
> I should have said that a trade optimizer
> not caring about positive definiteness is
> usually a bad thing -- it has its uses
> but giving the optimizer a shot at a
> portfolio with negative variance is asking
> for trouble (because it will do its best
> to get there).
>
> The estimates of the variance with only a
> few observations are going to be extremely
> noisy.  I think the length of history to
> be used to estimate the variance should
> depend on what length gives the best variance
> estimate and should be independent of the
> rebalancing schedule.  (Though perhaps I
> have misinterpreted the original post.)
>
> Pat
>
> On 22/09/2015 19:59, Patrick Burns wrote:
> > You can use a factor model or shrinkage
> > to get a positive definite variance matrix.
> > There is a function for each in the
> > BurStFin package on CRAN.
> >
> > The optimizer in Portfolio Probe doesn't
> > care about positive definiteness (though
> > that is not always a good thing).  It is
> > free for academic use.
> >
> > Pat
> >
> > On 22/09/2015 14:37, aschmid1 wrote:
> >> Hi everyone,
> >> I'm trying to estimate optimal Markowitz portfolio weights for a list of
> >> stocks chosen upon some criterion using solve.QP from quadprog library.
> >> When the number of stocks N reaches some limit, I get a message "matrix
> >> D in quadratic function is not positive definite." For example, if I
> >> rebalance every 6 weeks (which implies that variance is calculated for
> >> 6-week interval prior to the period for which I calculate portfolio
> >> weights), I can get solution for 25>=N<50. For 12-week interval,
> >> solution exists for 50>=N<100, and for 24-week interval, I can get
> >> solution for N=100. My attempt to remedy this problem with Higham's
> >> method doesn't help. I'll greatly appreciate you input: first, why this
> >> may happen (can there be lack of local minimum?), and second, whether
> >> there are R solvers that may need only semi positive definite matrix.
> >>
> >> Thanks! Alec
> >>
> >> _______________________________________________
> >> [hidden email] mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> >> -- Subscriber-posting only. If you want to post, subscribe first.
> >> -- Also note that this is not the r-help list where general R questions
> >> should go.
> >
>
> --
> Patrick Burns
> [hidden email]
> http://www.burns-stat.com
> http://www.portfolioprobe.com/blog
> twitter: @burnsstat @portfolioprobe
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-finance
> -- Subscriber-posting only. If you want to post, subscribe first.
> -- Also note that this is not the r-help list where general R questions
> should go.
>

        [[alternative HTML version deleted]]

_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-finance
-- Subscriber-posting only. If you want to post, subscribe first.
-- Also note that this is not the r-help list where general R questions should go.