portfolio.optim and error in solve.QP: matrix D not positive definite

7 messages
Open this post in threaded view
|

portfolio.optim and error in solve.QP: matrix D not positive definite

 Dear Group, I have  a large set of stocks and want to determine the efficient frontier. The data set covers approx. 1.5 years and S&P 500 companies (nothing weird). portfolio.optim from the PerformanceAnalytics package works very well and fast. However, whenever I decrease the number of stocks in the portfolio (to 10 or 400), I receive an error message: "solve.QP(Dmat, dvec, Amat, bvec = b0, meq = 2) :   matrix D in quadratic function is not positive definite!" My command settings for portfolio.optim were: seed <- portfolio.optim(t(x), pm = current_er, riskless = FALSE, shorts = FALSE, rf = 0.0) Even when I tried it with shorts = TRUE the error would still remain. x is the set of stocks (stocks in columns, time in rows), current_er is the target return (lies between the minimal mean and the maximum mean of a long only portfolio). I can not post the stock data here - so maybe you have some general suggestions for me of what could have gone wrong... The covariance matrix is positive definite. What could cause the problem? It works fine with the large data set but does not work at all with the small one... Thanks a lot for your suggestions! Lui _______________________________________________ [hidden email] mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance-- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
Open this post in threaded view
|

Re: portfolio.optim and error in solve.QP: matrix D not positive definite

 Hi Lui, I never worked with such kind of portfolio optimization problem but in Risk management practice it often comes as estimated VCV matrix is not a PD, hence it is not a truly VCV matrix. Root of this problem might be many, most importantly, it is incomplete and inconsistent return values. In such case, common practice is to disturb this estimated VCV matrix slightly, so that you would get **nearest** VCV matrix which is PD. Here you might be interested in: http://eprints.ma.man.ac.uk/232/01/covered/MIMS_ep2006_70.pdfTherefore I guess, what you need to do is perhaps debug the underlying codes and do some reverse-engineering to modify the underlying matrix to a nearest PD. HTH _____________________________________________________ Arun Kumar Saha, FRM QUANTITATIVE RISK AND HEDGE CONSULTING SPECIALIST Visit me at: http://in.linkedin.com/in/ArunFRM_____________________________________________________
Open this post in threaded view
|

Re: portfolio.optim and error in solve.QP: matrix D not positive definite

 In reply to this post by Lui ## Hi Lui, Without seeing the data this is just speculation but... Are you sure you want t(x)? If you're mixing up your observations versus your assets this may explain the error. The first parameter of portfolio.optim (in the tseries package) is a returns matrix, one column for each asset and one row for each day (assuming daily returns).  If you have this wrong then for your small datasets you'd have more columns than rows and this could produce that error. Also, you don't have to pass the entire returns matrix to portfolio.optim, you could pass just the covariance matrix you calculate yourself and a vector (1-row matrix) of mean returns as follows: library(tseries) set.seed(2) R <- matrix(rnorm(100*10),nrow=100,ncol=10) # 10 assets, 100 observations averet <- matrix(apply(R,2,mean),nrow=1) rcov <- cov(R) current_er <- 0.05 (op <- portfolio.optim(x=averet,pm=current_er,covmat=rcov,riskless = FALSE,shorts = FALSE, rf = 0.0)) Hope this helps. Best, Guy On 1/26/2011 7:51 PM, Lui ## wrote: > Dear Group, > > I have  a large set of stocks and want to determine the efficient > frontier. The data set covers approx. 1.5 years and S&P 500 companies > (nothing weird). portfolio.optim from the PerformanceAnalytics package > works very well and fast. However, whenever I decrease the number of > stocks in the portfolio (to 10 or 400), I receive an error message: > > "solve.QP(Dmat, dvec, Amat, bvec = b0, meq = 2) : >    matrix D in quadratic function is not positive definite!" > > My command settings for portfolio.optim were: > > seed<- portfolio.optim(t(x), pm = current_er, riskless = FALSE, > shorts = FALSE, rf = 0.0) > > Even when I tried it with shorts = TRUE the error would still remain. > x is the set of stocks (stocks in columns, time in rows), current_er > is the target return (lies between the minimal mean and the maximum > mean of a long only portfolio). > I can not post the stock data here - so maybe you have some general > suggestions for me of what could have gone wrong... The covariance > matrix is positive definite. What could cause the problem? It works > fine with the large data set but does not work at all with the small > one... > Thanks a lot for your suggestions! > > Lui > > _______________________________________________ > [hidden email] mailing list > https://stat.ethz.ch/mailman/listinfo/r-sig-finance> -- Subscriber-posting only. If you want to post, subscribe first. > -- Also note that this is not the r-help list where general R questions should go. > _______________________________________________ [hidden email] mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance-- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
Open this post in threaded view
|

Re: portfolio.optim and error in solve.QP: matrix D not positive definite

 Also the link below might help with a large number of assets this is a   common problem. https://stat.ethz.ch/pipermail/r-sig-finance/2008q3/002854.htmlCheers Krishna On Jan 27, 2011, at 12:03 PM, Guy Yollin <[hidden email]>   wrote: > Hi Lui, > > Without seeing the data this is just speculation but... > > Are you sure you want t(x)? If you're mixing up your observations   > versus your assets this may explain the error. > > The first parameter of portfolio.optim (in the tseries package) is a   > returns matrix, one column for each asset and one row for each day   > (assuming daily returns).  If you have this wrong then for your   > small datasets you'd have more columns than rows and this could   > produce that error. > > Also, you don't have to pass the entire returns matrix to   > portfolio.optim, you could pass just the covariance matrix you   > calculate yourself and a vector (1-row matrix) of mean returns as   > follows: > > > library(tseries) > set.seed(2) > R <- matrix(rnorm(100*10),nrow=100,ncol=10) # 10 assets, 100   > observations > averet <- matrix(apply(R,2,mean),nrow=1) > rcov <- cov(R) > current_er <- 0.05 > (op <- portfolio.optim(x=averet,pm=current_er,covmat=rcov,riskless =   > FALSE,shorts = FALSE, rf = 0.0)) > > Hope this helps. > > Best, > > Guy > > > On 1/26/2011 7:51 PM, Lui ## wrote: >> Dear Group, >> >> I have  a large set of stocks and want to determine the efficient >> frontier. The data set covers approx. 1.5 years and S&P 500 companies >> (nothing weird). portfolio.optim from the PerformanceAnalytics   >> package >> works very well and fast. However, whenever I decrease the number of >> stocks in the portfolio (to 10 or 400), I receive an error message: >> >> "solve.QP(Dmat, dvec, Amat, bvec = b0, meq = 2) : >>   matrix D in quadratic function is not positive definite!" >> >> My command settings for portfolio.optim were: >> >> seed<- portfolio.optim(t(x), pm = current_er, riskless = FALSE, >> shorts = FALSE, rf = 0.0) >> >> Even when I tried it with shorts = TRUE the error would still remain. >> x is the set of stocks (stocks in columns, time in rows), current_er >> is the target return (lies between the minimal mean and the maximum >> mean of a long only portfolio). >> I can not post the stock data here - so maybe you have some general >> suggestions for me of what could have gone wrong... The covariance >> matrix is positive definite. What could cause the problem? It works >> fine with the large data set but does not work at all with the small >> one... >> Thanks a lot for your suggestions! >> >> Lui >> >> _______________________________________________ >> [hidden email] mailing list >> https://stat.ethz.ch/mailman/listinfo/r-sig-finance>> -- Subscriber-posting only. If you want to post, subscribe first. >> -- Also note that this is not the r-help list where general R   >> questions should go. >> > > _______________________________________________ > [hidden email] mailing list > https://stat.ethz.ch/mailman/listinfo/r-sig-finance> -- Subscriber-posting only. If you want to post, subscribe first. > -- Also note that this is not the r-help list where general R   > questions should go. _______________________________________________ [hidden email] mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance-- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.
Open this post in threaded view
|

Re: portfolio.optim and error in solve.QP: matrix D not positive definite

 Hello everybody, sorry for my delayed "thanks" note - I was travelling. @Arun: Debugging the underlying code is a little bit difficult since the optimizer was written in FORTRAN. I think going for the nearest PD (as Krishna also suggested) might be the best way. However, I honestly don't understand why it is not PD... Does anybody have an explanation for that? @Guy: The weird thing is that I got the error code without the t(x) in the first place. t(x) solved the problem (for some assets) and the result indicated that it took a look at the assets and not the observations... I am going to give it a try with the covariance matrix again and let you know if it worked out... strange though. @Krishna: Thanks for your link! I think that really helps... I am going to try it out! Do you have an explanation why this is a common problem with a large number of assets? Thank you! Have a nice weekend! Lui On Fri, Jan 28, 2011 at 3:51 PM, krishna <[hidden email]> wrote: > Also the link below might help with a large number of assets this is a > common problem. > > https://stat.ethz.ch/pipermail/r-sig-finance/2008q3/002854.html> > > Cheers > Krishna > > > On Jan 27, 2011, at 12:03 PM, Guy Yollin <[hidden email]> wrote: > >> Hi Lui, >> >> Without seeing the data this is just speculation but... >> >> Are you sure you want t(x)? If you're mixing up your observations versus >> your assets this may explain the error. >> >> The first parameter of portfolio.optim (in the tseries package) is a >> returns matrix, one column for each asset and one row for each day (assuming >> daily returns).  If you have this wrong then for your small datasets you'd >> have more columns than rows and this could produce that error. >> >> Also, you don't have to pass the entire returns matrix to portfolio.optim, >> you could pass just the covariance matrix you calculate yourself and a >> vector (1-row matrix) of mean returns as follows: >> >> >> library(tseries) >> set.seed(2) >> R <- matrix(rnorm(100*10),nrow=100,ncol=10) # 10 assets, 100 observations >> averet <- matrix(apply(R,2,mean),nrow=1) >> rcov <- cov(R) >> current_er <- 0.05 >> (op <- portfolio.optim(x=averet,pm=current_er,covmat=rcov,riskless = >> FALSE,shorts = FALSE, rf = 0.0)) >> >> Hope this helps. >> >> Best, >> >> Guy >> >> >> On 1/26/2011 7:51 PM, Lui ## wrote: >>> >>> Dear Group, >>> >>> I have  a large set of stocks and want to determine the efficient >>> frontier. The data set covers approx. 1.5 years and S&P 500 companies >>> (nothing weird). portfolio.optim from the PerformanceAnalytics package >>> works very well and fast. However, whenever I decrease the number of >>> stocks in the portfolio (to 10 or 400), I receive an error message: >>> >>> "solve.QP(Dmat, dvec, Amat, bvec = b0, meq = 2) : >>>  matrix D in quadratic function is not positive definite!" >>> >>> My command settings for portfolio.optim were: >>> >>> seed<- portfolio.optim(t(x), pm = current_er, riskless = FALSE, >>> shorts = FALSE, rf = 0.0) >>> >>> Even when I tried it with shorts = TRUE the error would still remain. >>> x is the set of stocks (stocks in columns, time in rows), current_er >>> is the target return (lies between the minimal mean and the maximum >>> mean of a long only portfolio). >>> I can not post the stock data here - so maybe you have some general >>> suggestions for me of what could have gone wrong... The covariance >>> matrix is positive definite. What could cause the problem? It works >>> fine with the large data set but does not work at all with the small >>> one... >>> Thanks a lot for your suggestions! >>> >>> Lui >>> >>> _______________________________________________ >>> [hidden email] mailing list >>> https://stat.ethz.ch/mailman/listinfo/r-sig-finance>>> -- Subscriber-posting only. If you want to post, subscribe first. >>> -- Also note that this is not the r-help list where general R questions >>> should go. >>> >> >> _______________________________________________ >> [hidden email] mailing list >> https://stat.ethz.ch/mailman/listinfo/r-sig-finance>> -- Subscriber-posting only. If you want to post, subscribe first. >> -- Also note that this is not the r-help list where general R questions >> should go. > > _______________________________________________ > [hidden email] mailing list > https://stat.ethz.ch/mailman/listinfo/r-sig-finance> -- Subscriber-posting only. If you want to post, subscribe first. > -- Also note that this is not the r-help list where general R questions > should go. > _______________________________________________ [hidden email] mailing list https://stat.ethz.ch/mailman/listinfo/r-sig-finance-- Subscriber-posting only. If you want to post, subscribe first. -- Also note that this is not the r-help list where general R questions should go.