

Hello everybody,
I have been using the random_portfolios function from the
`PortfolioAnalytics` package to simulate the range of possibilities for
return paths at each step under various portfolio constraints / mandates
for evaluating mutual fund managers. As more managers are added to the
universe, however, and more simulations are needed, the pure R
implementations get pretty heavy and hard to scale. I was wondering if
there has been any work out there thus far on implementing any of the three
random portfolio generation methods (sample, simplex, and grid search) at a
lower level, using something like `Rcpp` to enhance the efficiency of these
algorithms?
Any help/feedback is much appreciated.
Thank you,

Kshitij Dhingra
Applied Academics LLC
Office: +1.917.262.0516
Mobile: +1.206.696.5945
Email: [hidden email]
Website: http://www.AppliedAcademics.com [[alternative HTML version deleted]]
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


On Mon, 20170320 at 15:09 0400, Kevin Dhingra wrote:
> I have been using the random_portfolios function from the
> `PortfolioAnalytics` package to simulate the range of possibilities
> for return paths at each step under various portfolio constraints /
> mandates for evaluating mutual fund managers. As more managers are
> added to the universe, however, and more simulations are needed, the
> pure R implementations get pretty heavy and hard to scale. I was
> wondering if there has been any work out there thus far on
> implementing any of the three random portfolio generation methods
> (sample, simplex, and grid search) at a lower level, using something
> like `Rcpp` to enhance the efficiency of these algorithms?
We've discussed it, but I can't say that it is terribly high on our
list of priorities.
In most cases, no more than 12k portfolios should be required to get a
fair view of the feasible space given your constraints and objectives.
We'd be happy to work with you if you want to craft a patch to use C or
Rcpp for this.
Regards,
Brian
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


Brian,
Thank you for a quick reply. I will soon be working on that problem and
from what I have played with so far, it is unlikely that for our example
~2k portfolios will be enough (really hoping it would) to get a good sense
of the feasible space and seems like I need to implement an Rcpp version of
the random portfolios function. I will be happy to collaborate and share my
code once i get a decent handle on it locally for the purposes of our
current project.
Regards,
Kshitij Dhingra
On Mon, Mar 20, 2017 at 3:17 PM, Brian G. Peterson < [hidden email]>
wrote:
> On Mon, 20170320 at 15:09 0400, Kevin Dhingra wrote:
> > I have been using the random_portfolios function from the
> > `PortfolioAnalytics` package to simulate the range of possibilities
> > for return paths at each step under various portfolio constraints /
> > mandates for evaluating mutual fund managers. As more managers are
> > added to the universe, however, and more simulations are needed, the
> > pure R implementations get pretty heavy and hard to scale. I was
> > wondering if there has been any work out there thus far on
> > implementing any of the three random portfolio generation methods
> > (sample, simplex, and grid search) at a lower level, using something
> > like `Rcpp` to enhance the efficiency of these algorithms?
>
> We've discussed it, but I can't say that it is terribly high on our
> list of priorities.
>
> In most cases, no more than 12k portfolios should be required to get a
> fair view of the feasible space given your constraints and objectives.
>
> We'd be happy to work with you if you want to craft a patch to use C or
> Rcpp for this.
>
> Regards,
>
> Brian
>

Kshitij Dhingra
Applied Academics LLC
Office: +1.917.262.0516
Mobile: +1.206.696.5945
Email: [hidden email]
Website: http://www.AppliedAcademics.com [[alternative HTML version deleted]]
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


Kevin,
Can you give us a sense of the number of assets in the portfolio and
the constraints? That will help us understand where the potential
bottlenecks are in the random portfolio generation. For example,
generating a set of random portfolios for box and weight constraints
if relatively fast, but adding group or position limit constraints
makes the algorithm more complicated and slower.
Thanks,
Ross
On Mon, Mar 20, 2017 at 2:35 PM, Kevin Dhingra
< [hidden email]> wrote:
> Brian,
>
> Thank you for a quick reply. I will soon be working on that problem and
> from what I have played with so far, it is unlikely that for our example
> ~2k portfolios will be enough (really hoping it would) to get a good sense
> of the feasible space and seems like I need to implement an Rcpp version of
> the random portfolios function. I will be happy to collaborate and share my
> code once i get a decent handle on it locally for the purposes of our
> current project.
>
> Regards,
> Kshitij Dhingra
>
>
>
> On Mon, Mar 20, 2017 at 3:17 PM, Brian G. Peterson < [hidden email]>
> wrote:
>
>> On Mon, 20170320 at 15:09 0400, Kevin Dhingra wrote:
>> > I have been using the random_portfolios function from the
>> > `PortfolioAnalytics` package to simulate the range of possibilities
>> > for return paths at each step under various portfolio constraints /
>> > mandates for evaluating mutual fund managers. As more managers are
>> > added to the universe, however, and more simulations are needed, the
>> > pure R implementations get pretty heavy and hard to scale. I was
>> > wondering if there has been any work out there thus far on
>> > implementing any of the three random portfolio generation methods
>> > (sample, simplex, and grid search) at a lower level, using something
>> > like `Rcpp` to enhance the efficiency of these algorithms?
>>
>> We've discussed it, but I can't say that it is terribly high on our
>> list of priorities.
>>
>> In most cases, no more than 12k portfolios should be required to get a
>> fair view of the feasible space given your constraints and objectives.
>>
>> We'd be happy to work with you if you want to craft a patch to use C or
>> Rcpp for this.
>>
>> Regards,
>>
>> Brian
>>
>
>
>
> 
> Kshitij Dhingra
> Applied Academics LLC
> Office: +1.917.262.0516
> Mobile: +1.206.696.5945
> Email: [hidden email]
> Website: http://www.AppliedAcademics.com>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/rsigfinance>  Subscriberposting only. If you want to post, subscribe first.
>  Also note that this is not the rhelp list where general R questions should go.
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


Hi Ross,
Sure. Even though I have not profiled the bottlenecks quite in detail as of
yet, i will give you a decent idea of the problem I am working with. I can
have multiple indices with as much as 2000 assets with group, position and
turnover limits (Not sure if i can increase the speed by removing
constraints and doing rejection sampling later). In order to generate a
daily possible set for the market in this case, I was playing around with
~45 thousand permutations. Also I think I will end up using the "sample"
method because of the type of constraints we have and as you already have
mentioned that method is the slowest (takes about 30 times the time using
"simplex" for the same constraints). Adding box and position limit
constraints are causing it to run a bit slower (but its not a big
difference). I can always provide a more thorough analysis of the potential
bottlenecks with a lot more detail when I have a chance to start working on
translating it to cpp
Thank you,
On Mon, Mar 20, 2017 at 4:04 PM, Ross Bennett < [hidden email]>
wrote:
> Kevin,
>
> Can you give us a sense of the number of assets in the portfolio and
> the constraints? That will help us understand where the potential
> bottlenecks are in the random portfolio generation. For example,
> generating a set of random portfolios for box and weight constraints
> if relatively fast, but adding group or position limit constraints
> makes the algorithm more complicated and slower.
>
> Thanks,
> Ross
>
>
> On Mon, Mar 20, 2017 at 2:35 PM, Kevin Dhingra
> < [hidden email]> wrote:
> > Brian,
> >
> > Thank you for a quick reply. I will soon be working on that problem and
> > from what I have played with so far, it is unlikely that for our example
> > ~2k portfolios will be enough (really hoping it would) to get a good
> sense
> > of the feasible space and seems like I need to implement an Rcpp version
> of
> > the random portfolios function. I will be happy to collaborate and share
> my
> > code once i get a decent handle on it locally for the purposes of our
> > current project.
> >
> > Regards,
> > Kshitij Dhingra
> >
> >
> >
> > On Mon, Mar 20, 2017 at 3:17 PM, Brian G. Peterson < [hidden email]>
> > wrote:
> >
> >> On Mon, 20170320 at 15:09 0400, Kevin Dhingra wrote:
> >> > I have been using the random_portfolios function from the
> >> > `PortfolioAnalytics` package to simulate the range of possibilities
> >> > for return paths at each step under various portfolio constraints /
> >> > mandates for evaluating mutual fund managers. As more managers are
> >> > added to the universe, however, and more simulations are needed, the
> >> > pure R implementations get pretty heavy and hard to scale. I was
> >> > wondering if there has been any work out there thus far on
> >> > implementing any of the three random portfolio generation methods
> >> > (sample, simplex, and grid search) at a lower level, using something
> >> > like `Rcpp` to enhance the efficiency of these algorithms?
> >>
> >> We've discussed it, but I can't say that it is terribly high on our
> >> list of priorities.
> >>
> >> In most cases, no more than 12k portfolios should be required to get a
> >> fair view of the feasible space given your constraints and objectives.
> >>
> >> We'd be happy to work with you if you want to craft a patch to use C or
> >> Rcpp for this.
> >>
> >> Regards,
> >>
> >> Brian
> >>
> >
> >
> >
> > 
> > Kshitij Dhingra
> > Applied Academics LLC
> > Office: +1.917.262.0516
> > Mobile: +1.206.696.5945
> > Email: [hidden email]
> > Website: http://www.AppliedAcademics.com> >
> > [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > [hidden email] mailing list
> > https://stat.ethz.ch/mailman/listinfo/rsigfinance> >  Subscriberposting only. If you want to post, subscribe first.
> >  Also note that this is not the rhelp list where general R questions
> should go.
>

Kshitij Dhingra
Applied Academics LLC
Office: +1.917.262.0516
Mobile: +1.206.696.5945
Email: [hidden email]
Website: http://www.AppliedAcademics.com [[alternative HTML version deleted]]
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


For this type of problem, I would probably generate one set of random
portfolios and just reuse that set of feasible portfolios... My usual
rule is nassets + 12k feasible portfolios. You can get a better
number e.g. from sampling theory, but this should be enough.
Once you have this weights matrix rp, you only need to regenerate rp if
your universe changes.
Still interested in a more efficient implementation, of course, or we
can work with you to see if we can find resources to work on it, e.g.
from academia.
Regards,
Brian
On 03/20/2017 05:28 PM, Kevin Dhingra wrote:
> Hi Ross,
>
> Sure. Even though I have not profiled the bottlenecks quite in detail as of
> yet, i will give you a decent idea of the problem I am working with. I can
> have multiple indices with as much as 2000 assets with group, position and
> turnover limits (Not sure if i can increase the speed by removing
> constraints and doing rejection sampling later). In order to generate a
> daily possible set for the market in this case, I was playing around with
> ~45 thousand permutations. Also I think I will end up using the "sample"
> method because of the type of constraints we have and as you already have
> mentioned that method is the slowest (takes about 30 times the time using
> "simplex" for the same constraints). Adding box and position limit
> constraints are causing it to run a bit slower (but its not a big
> difference). I can always provide a more thorough analysis of the potential
> bottlenecks with a lot more detail when I have a chance to start working on
> translating it to cpp
>
> Thank you,
>
> On Mon, Mar 20, 2017 at 4:04 PM, Ross Bennett < [hidden email]>
> wrote:
>
>> Kevin,
>>
>> Can you give us a sense of the number of assets in the portfolio and
>> the constraints? That will help us understand where the potential
>> bottlenecks are in the random portfolio generation. For example,
>> generating a set of random portfolios for box and weight constraints
>> if relatively fast, but adding group or position limit constraints
>> makes the algorithm more complicated and slower.
>>
>> Thanks,
>> Ross
>>
>>
>> On Mon, Mar 20, 2017 at 2:35 PM, Kevin Dhingra
>> < [hidden email]> wrote:
>>> Brian,
>>>
>>> Thank you for a quick reply. I will soon be working on that problem and
>>> from what I have played with so far, it is unlikely that for our example
>>> ~2k portfolios will be enough (really hoping it would) to get a good
>> sense
>>> of the feasible space and seems like I need to implement an Rcpp version
>> of
>>> the random portfolios function. I will be happy to collaborate and share
>> my
>>> code once i get a decent handle on it locally for the purposes of our
>>> current project.
>>>
>>> Regards,
>>> Kshitij Dhingra
>>>
>>>
>>>
>>> On Mon, Mar 20, 2017 at 3:17 PM, Brian G. Peterson < [hidden email]>
>>> wrote:
>>>
>>>> On Mon, 20170320 at 15:09 0400, Kevin Dhingra wrote:
>>>>> I have been using the random_portfolios function from the
>>>>> `PortfolioAnalytics` package to simulate the range of possibilities
>>>>> for return paths at each step under various portfolio constraints /
>>>>> mandates for evaluating mutual fund managers. As more managers are
>>>>> added to the universe, however, and more simulations are needed, the
>>>>> pure R implementations get pretty heavy and hard to scale. I was
>>>>> wondering if there has been any work out there thus far on
>>>>> implementing any of the three random portfolio generation methods
>>>>> (sample, simplex, and grid search) at a lower level, using something
>>>>> like `Rcpp` to enhance the efficiency of these algorithms?
>>>>
>>>> We've discussed it, but I can't say that it is terribly high on our
>>>> list of priorities.
>>>>
>>>> In most cases, no more than 12k portfolios should be required to get a
>>>> fair view of the feasible space given your constraints and objectives.
>>>>
>>>> We'd be happy to work with you if you want to craft a patch to use C or
>>>> Rcpp for this.
>>>>
>>>> Regards,
>>>>
>>>> Brian
>>>>
>>>
>>>
>>>
>>> 
>>> Kshitij Dhingra
>>> Applied Academics LLC
>>> Office: +1.917.262.0516
>>> Mobile: +1.206.696.5945
>>> Email: [hidden email]
>>> Website: http://www.AppliedAcademics.com>>>
>>> [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> [hidden email] mailing list
>>> https://stat.ethz.ch/mailman/listinfo/rsigfinance>>>  Subscriberposting only. If you want to post, subscribe first.
>>>  Also note that this is not the rhelp list where general R questions
>> should go.
>>
>
>
>

Brian G. Peterson
http://braverock.com/brian/Ph: 7734594973
IM: bgpbraverock
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


Brian,
Yes I think that will be a good starting point. My universe would not
change a lot (I will be working with 1015 benchmarks at a time and I guess
I can generate a reusable set for each independently before running it
through my main algorithm). Having said that, I envision the investment
mandates/constraints changing quite a lot (both in the cross section and
also over time for the same manager). I am hoping there must be a way
around it using rejection sampling but have not done enough research to
comment on how that solution works for such big dimensions. It will be
really helpful if you could point me to any specific resources from
academia for the same (Haven't been able to find much about random
portfolios myself except Portfolio Analytics and Patrick Burns work on
Portfolio Probe). As a side note  Do you think translating it using Rcpp
would be time well spent or you think there must be a smarter way to get
around it still using R?
I really appreciate your help on this thread.
Regards,
Kshitij Dhingra
On Mon, Mar 20, 2017 at 7:21 PM, Brian G. Peterson < [hidden email]>
wrote:
> For this type of problem, I would probably generate one set of random
> portfolios and just reuse that set of feasible portfolios... My usual rule
> is nassets + 12k feasible portfolios. You can get a better number e.g.
> from sampling theory, but this should be enough.
>
> Once you have this weights matrix rp, you only need to regenerate rp if
> your universe changes.
>
> Still interested in a more efficient implementation, of course, or we can
> work with you to see if we can find resources to work on it, e.g. from
> academia.
>
> Regards,
>
> Brian
>
>
> On 03/20/2017 05:28 PM, Kevin Dhingra wrote:
>
>> Hi Ross,
>>
>> Sure. Even though I have not profiled the bottlenecks quite in detail as
>> of
>> yet, i will give you a decent idea of the problem I am working with. I can
>> have multiple indices with as much as 2000 assets with group, position and
>> turnover limits (Not sure if i can increase the speed by removing
>> constraints and doing rejection sampling later). In order to generate a
>> daily possible set for the market in this case, I was playing around with
>> ~45 thousand permutations. Also I think I will end up using the "sample"
>> method because of the type of constraints we have and as you already have
>> mentioned that method is the slowest (takes about 30 times the time using
>> "simplex" for the same constraints). Adding box and position limit
>> constraints are causing it to run a bit slower (but its not a big
>> difference). I can always provide a more thorough analysis of the
>> potential
>> bottlenecks with a lot more detail when I have a chance to start working
>> on
>> translating it to cpp
>>
>> Thank you,
>>
>> On Mon, Mar 20, 2017 at 4:04 PM, Ross Bennett < [hidden email]>
>> wrote:
>>
>> Kevin,
>>>
>>> Can you give us a sense of the number of assets in the portfolio and
>>> the constraints? That will help us understand where the potential
>>> bottlenecks are in the random portfolio generation. For example,
>>> generating a set of random portfolios for box and weight constraints
>>> if relatively fast, but adding group or position limit constraints
>>> makes the algorithm more complicated and slower.
>>>
>>> Thanks,
>>> Ross
>>>
>>>
>>> On Mon, Mar 20, 2017 at 2:35 PM, Kevin Dhingra
>>> < [hidden email]> wrote:
>>>
>>>> Brian,
>>>>
>>>> Thank you for a quick reply. I will soon be working on that problem and
>>>> from what I have played with so far, it is unlikely that for our example
>>>> ~2k portfolios will be enough (really hoping it would) to get a good
>>>>
>>> sense
>>>
>>>> of the feasible space and seems like I need to implement an Rcpp version
>>>>
>>> of
>>>
>>>> the random portfolios function. I will be happy to collaborate and share
>>>>
>>> my
>>>
>>>> code once i get a decent handle on it locally for the purposes of our
>>>> current project.
>>>>
>>>> Regards,
>>>> Kshitij Dhingra
>>>>
>>>>
>>>>
>>>> On Mon, Mar 20, 2017 at 3:17 PM, Brian G. Peterson < [hidden email]
>>>> >
>>>> wrote:
>>>>
>>>> On Mon, 20170320 at 15:09 0400, Kevin Dhingra wrote:
>>>>>
>>>>>> I have been using the random_portfolios function from the
>>>>>> `PortfolioAnalytics` package to simulate the range of possibilities
>>>>>> for return paths at each step under various portfolio constraints /
>>>>>> mandates for evaluating mutual fund managers. As more managers are
>>>>>> added to the universe, however, and more simulations are needed, the
>>>>>> pure R implementations get pretty heavy and hard to scale. I was
>>>>>> wondering if there has been any work out there thus far on
>>>>>> implementing any of the three random portfolio generation methods
>>>>>> (sample, simplex, and grid search) at a lower level, using something
>>>>>> like `Rcpp` to enhance the efficiency of these algorithms?
>>>>>>
>>>>>
>>>>> We've discussed it, but I can't say that it is terribly high on our
>>>>> list of priorities.
>>>>>
>>>>> In most cases, no more than 12k portfolios should be required to get a
>>>>> fair view of the feasible space given your constraints and objectives.
>>>>>
>>>>> We'd be happy to work with you if you want to craft a patch to use C or
>>>>> Rcpp for this.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Brian
>>>>>
>>>>>
>>>>
>>>>
>>>> 
>>>> Kshitij Dhingra
>>>> Applied Academics LLC
>>>> Office: +1.917.262.0516
>>>> Mobile: +1.206.696.5945
>>>> Email: [hidden email]
>>>> Website: http://www.AppliedAcademics.com>>>>
>>>> [[alternative HTML version deleted]]
>>>>
>>>> _______________________________________________
>>>> [hidden email] mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/rsigfinance>>>>  Subscriberposting only. If you want to post, subscribe first.
>>>>  Also note that this is not the rhelp list where general R questions
>>>>
>>> should go.
>>>
>>>
>>
>>
>>
>
> 
> Brian G. Peterson
> http://braverock.com/brian/> Ph: 7734594973
> IM: bgpbraverock
>
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/rsigfinance>  Subscriberposting only. If you want to post, subscribe first.
>  Also note that this is not the rhelp list where general R questions
> should go.
>

Kshitij Dhingra
Applied Academics LLC
Office: +1.917.262.0516
Mobile: +1.206.696.5945
Email: [hidden email]
Website: http://www.AppliedAcademics.com [[alternative HTML version deleted]]
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


Please look at my package rportfolios. It allows you to construct individual and samples of random portfolios that satisfy a wide range of constraints ( e.g., long only, short only, long short portfolios with notional exposure constraints, bounded weights).
Regards,
Frederick Novomestky
Original Message
>From: Kevin Dhingra < [hidden email]>
>Sent: Mar 20, 2017 6:28 PM
>To: Ross Bennett < [hidden email]>
>Cc: " [hidden email]" < [hidden email]>
>Subject: Re: [RSIGFinance] random portfolios
>
>Hi Ross,
>
>Sure. Even though I have not profiled the bottlenecks quite in detail as of
>yet, i will give you a decent idea of the problem I am working with. I can
>have multiple indices with as much as 2000 assets with group, position and
>turnover limits (Not sure if i can increase the speed by removing
>constraints and doing rejection sampling later). In order to generate a
>daily possible set for the market in this case, I was playing around with
>~45 thousand permutations. Also I think I will end up using the "sample"
>method because of the type of constraints we have and as you already have
>mentioned that method is the slowest (takes about 30 times the time using
>"simplex" for the same constraints). Adding box and position limit
>constraints are causing it to run a bit slower (but its not a big
>difference). I can always provide a more thorough analysis of the potential
>bottlenecks with a lot more detail when I have a chance to start working on
>translating it to cpp
>
>Thank you,
>
>On Mon, Mar 20, 2017 at 4:04 PM, Ross Bennett < [hidden email]>
>wrote:
>
>> Kevin,
>>
>> Can you give us a sense of the number of assets in the portfolio and
>> the constraints? That will help us understand where the potential
>> bottlenecks are in the random portfolio generation. For example,
>> generating a set of random portfolios for box and weight constraints
>> if relatively fast, but adding group or position limit constraints
>> makes the algorithm more complicated and slower.
>>
>> Thanks,
>> Ross
>>
>>
>> On Mon, Mar 20, 2017 at 2:35 PM, Kevin Dhingra
>> < [hidden email]> wrote:
>> > Brian,
>> >
>> > Thank you for a quick reply. I will soon be working on that problem and
>> > from what I have played with so far, it is unlikely that for our example
>> > ~2k portfolios will be enough (really hoping it would) to get a good
>> sense
>> > of the feasible space and seems like I need to implement an Rcpp version
>> of
>> > the random portfolios function. I will be happy to collaborate and share
>> my
>> > code once i get a decent handle on it locally for the purposes of our
>> > current project.
>> >
>> > Regards,
>> > Kshitij Dhingra
>> >
>> >
>> >
>> > On Mon, Mar 20, 2017 at 3:17 PM, Brian G. Peterson < [hidden email]>
>> > wrote:
>> >
>> >> On Mon, 20170320 at 15:09 0400, Kevin Dhingra wrote:
>> >> > I have been using the random_portfolios function from the
>> >> > `PortfolioAnalytics` package to simulate the range of possibilities
>> >> > for return paths at each step under various portfolio constraints /
>> >> > mandates for evaluating mutual fund managers. As more managers are
>> >> > added to the universe, however, and more simulations are needed, the
>> >> > pure R implementations get pretty heavy and hard to scale. I was
>> >> > wondering if there has been any work out there thus far on
>> >> > implementing any of the three random portfolio generation methods
>> >> > (sample, simplex, and grid search) at a lower level, using something
>> >> > like `Rcpp` to enhance the efficiency of these algorithms?
>> >>
>> >> We've discussed it, but I can't say that it is terribly high on our
>> >> list of priorities.
>> >>
>> >> In most cases, no more than 12k portfolios should be required to get a
>> >> fair view of the feasible space given your constraints and objectives.
>> >>
>> >> We'd be happy to work with you if you want to craft a patch to use C or
>> >> Rcpp for this.
>> >>
>> >> Regards,
>> >>
>> >> Brian
>> >>
>> >
>> >
>> >
>> > 
>> > Kshitij Dhingra
>> > Applied Academics LLC
>> > Office: +1.917.262.0516
>> > Mobile: +1.206.696.5945
>> > Email: [hidden email]
>> > Website: http://www.AppliedAcademics.com>> >
>> > [[alternative HTML version deleted]]
>> >
>> > _______________________________________________
>> > [hidden email] mailing list
>> > https://stat.ethz.ch/mailman/listinfo/rsigfinance>> >  Subscriberposting only. If you want to post, subscribe first.
>> >  Also note that this is not the rhelp list where general R questions
>> should go.
>>
>
>
>
>
>Kshitij Dhingra
>Applied Academics LLC
>Office: +1.917.262.0516
>Mobile: +1.206.696.5945
>Email: [hidden email]
>Website: http://www.AppliedAcademics.com>
> [[alternative HTML version deleted]]
>
>_______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/rsigfinance> Subscriberposting only. If you want to post, subscribe first.
> Also note that this is not the rhelp list where general R questions should go.
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


The process you describe is pretty standard for an investmentcommittee
driven process.
I'm going to suggest that you don't really want to change the
constraints that often. For example, box constraints should be as large
as your overall investment mandate allows to give you the greatest
possible room for allocations. Sector or Factor constraints likewise
should be as minimal as possible just to guarantee the degree of
diversification described in your investment mandate.
The reason I'm suggesting this minimal constraint set is one of the
reasons we wrote the random portfolio code in the first place. To see
what I mean, generate a set of unconstrained random portfolios (or e.g.
only with a fullinvestment constraint). Then generate sets of
constrained random portfolios, adding your various constraint sets.
Plot the different sets on the same risk/return scatter plot, using
different colors for each set. Note how small the feasible space
becomes, very quickly.
This shrinkage of the feasible space has some good shrinkage
properties... moderate shrinkage actually decreases the possible impact
of estimation error in the various inputs, a little. Large amounts of
shrinkage (overly restrictive constraints) will do the opposite, and
magnify the negative out of sample impact of estimation error.
The academic literature mostly focuses on analytical solvers (e.g.
quadratic, linear, etc) and simple constraint sets. We've cited papers
by Patrick Burns as well as papers on the simplex models in
PortfolioAnalytics, but the literature is not vast.
Numerical solvers become important as the feasible space becomes
nonsmooth. One of the things that can create a nonsmooth feasible
space is a complex, overlapping constraint set.
The rportfolios package proposed by Frederick Novomestky also seems to
be an Ronly implementation, at a glance relying on truncated random
binomial vectors rather than truncated random uniform vectors. I
believe it will have similar performance characteristics to the
Burnsstyle random sample portfolios, and it seems to support fewer
constraint sets (no overlapping sector, group, or factor constraints
that I see). In any case, it generates matrices of weights that are
likely compatible with the PortfolioAnalytics random or seed portfolio
inputs. So if it works for you, that's great.
You also discuss using rejection after generating the portfolios. This
is the method used internally by random.portfolios to reject individual
weights if a constraint is violated. I'll have to evaluate whether the
truncdist package used by rportfolios could be more efficient than the
runif that is used by the current code. PortfolioAnalytics also allows
portfolios to be penalized in the solver, so that more complex cases can
be considered, or interactions between constraints and objectives.
To answer the question of whether Rcpp will help is somewhat complex.
I'm confident that some of the nested loops in the generation code will
be sped up by Rcpp. It is possible that more efficient algorithms are
available for constructing the weight vectors. A reason that this
hasn't been a huge priority though is that construction of the random
portfolio matrix is usually not the time limiter in a large
optimization: your objective function is. I think it will be possible
to improve the efficiency of this step, though it is unclear how much of
an impact this should have in practice to a large and complicated
numerically solved portfolio optimization problem.
Regards,
Brian
On 03/20/2017 07:06 PM, Kevin Dhingra wrote:
> Brian,
>
> Yes I think that will be a good starting point. My universe would not
> change a lot (I will be working with 1015 benchmarks at a time and I guess
> I can generate a reusable set for each independently before running it
> through my main algorithm). Having said that, I envision the investment
> mandates/constraints changing quite a lot (both in the cross section and
> also over time for the same manager). I am hoping there must be a way
> around it using rejection sampling but have not done enough research to
> comment on how that solution works for such big dimensions. It will be
> really helpful if you could point me to any specific resources from
> academia for the same (Haven't been able to find much about random
> portfolios myself except Portfolio Analytics and Patrick Burns work on
> Portfolio Probe). As a side note  Do you think translating it using Rcpp
> would be time well spent or you think there must be a smarter way to get
> around it still using R?
>
> I really appreciate your help on this thread.
>
> Regards,
> Kshitij Dhingra
>
> On Mon, Mar 20, 2017 at 7:21 PM, Brian G. Peterson < [hidden email]>
> wrote:
>
>> For this type of problem, I would probably generate one set of random
>> portfolios and just reuse that set of feasible portfolios... My usual rule
>> is nassets + 12k feasible portfolios. You can get a better number e.g.
>> from sampling theory, but this should be enough.
>>
>> Once you have this weights matrix rp, you only need to regenerate rp if
>> your universe changes.
>>
>> Still interested in a more efficient implementation, of course, or we can
>> work with you to see if we can find resources to work on it, e.g. from
>> academia.
>>
>> Regards,
>>
>> Brian
>>
>>
>> On 03/20/2017 05:28 PM, Kevin Dhingra wrote:
>>
>>> Hi Ross,
>>>
>>> Sure. Even though I have not profiled the bottlenecks quite in detail as
>>> of
>>> yet, i will give you a decent idea of the problem I am working with. I can
>>> have multiple indices with as much as 2000 assets with group, position and
>>> turnover limits (Not sure if i can increase the speed by removing
>>> constraints and doing rejection sampling later). In order to generate a
>>> daily possible set for the market in this case, I was playing around with
>>> ~45 thousand permutations. Also I think I will end up using the "sample"
>>> method because of the type of constraints we have and as you already have
>>> mentioned that method is the slowest (takes about 30 times the time using
>>> "simplex" for the same constraints). Adding box and position limit
>>> constraints are causing it to run a bit slower (but its not a big
>>> difference). I can always provide a more thorough analysis of the
>>> potential
>>> bottlenecks with a lot more detail when I have a chance to start working
>>> on
>>> translating it to cpp
>>>
>>> Thank you,
>>>
>>> On Mon, Mar 20, 2017 at 4:04 PM, Ross Bennett < [hidden email]>
>>> wrote:
>>>
>>> Kevin,
>>>>
>>>> Can you give us a sense of the number of assets in the portfolio and
>>>> the constraints? That will help us understand where the potential
>>>> bottlenecks are in the random portfolio generation. For example,
>>>> generating a set of random portfolios for box and weight constraints
>>>> if relatively fast, but adding group or position limit constraints
>>>> makes the algorithm more complicated and slower.
>>>>
>>>> Thanks,
>>>> Ross
>>>>
>>>>
>>>> On Mon, Mar 20, 2017 at 2:35 PM, Kevin Dhingra
>>>> < [hidden email]> wrote:
>>>>
>>>>> Brian,
>>>>>
>>>>> Thank you for a quick reply. I will soon be working on that problem and
>>>>> from what I have played with so far, it is unlikely that for our example
>>>>> ~2k portfolios will be enough (really hoping it would) to get a good
>>>>>
>>>> sense
>>>>
>>>>> of the feasible space and seems like I need to implement an Rcpp version
>>>>>
>>>> of
>>>>
>>>>> the random portfolios function. I will be happy to collaborate and share
>>>>>
>>>> my
>>>>
>>>>> code once i get a decent handle on it locally for the purposes of our
>>>>> current project.
>>>>>
>>>>> Regards,
>>>>> Kshitij Dhingra
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Mar 20, 2017 at 3:17 PM, Brian G. Peterson < [hidden email]
>>>>>>
>>>>> wrote:
>>>>>
>>>>> On Mon, 20170320 at 15:09 0400, Kevin Dhingra wrote:
>>>>>>
>>>>>>> I have been using the random_portfolios function from the
>>>>>>> `PortfolioAnalytics` package to simulate the range of possibilities
>>>>>>> for return paths at each step under various portfolio constraints /
>>>>>>> mandates for evaluating mutual fund managers. As more managers are
>>>>>>> added to the universe, however, and more simulations are needed, the
>>>>>>> pure R implementations get pretty heavy and hard to scale. I was
>>>>>>> wondering if there has been any work out there thus far on
>>>>>>> implementing any of the three random portfolio generation methods
>>>>>>> (sample, simplex, and grid search) at a lower level, using something
>>>>>>> like `Rcpp` to enhance the efficiency of these algorithms?
>>>>>>>
>>>>>>
>>>>>> We've discussed it, but I can't say that it is terribly high on our
>>>>>> list of priorities.
>>>>>>
>>>>>> In most cases, no more than 12k portfolios should be required to get a
>>>>>> fair view of the feasible space given your constraints and objectives.
>>>>>>
>>>>>> We'd be happy to work with you if you want to craft a patch to use C or
>>>>>> Rcpp for this.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Brian
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> 
>>>>> Kshitij Dhingra
>>>>> Applied Academics LLC
>>>>> Office: +1.917.262.0516
>>>>> Mobile: +1.206.696.5945
>>>>> Email: [hidden email]
>>>>> Website: http://www.AppliedAcademics.com>>>>>
>>>>> [[alternative HTML version deleted]]
>>>>>
>>>>> _______________________________________________
>>>>> [hidden email] mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/rsigfinance>>>>>  Subscriberposting only. If you want to post, subscribe first.
>>>>>  Also note that this is not the rhelp list where general R questions
>>>>>
>>>> should go.
>>>>
>>>>
>>>
>>>
>>>
>>
>> 
>> Brian G. Peterson
>> http://braverock.com/brian/>> Ph: 7734594973
>> IM: bgpbraverock
>>
>>
>> _______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/rsigfinance>>  Subscriberposting only. If you want to post, subscribe first.
>>  Also note that this is not the rhelp list where general R questions
>> should go.
>>
>
>
>

Brian G. Peterson
http://braverock.com/brian/Ph: 7734594973
IM: bgpbraverock
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


Just wanted to point out an example of what Brian mentioned for
visualizing the feasible space with different constraint sets.
Visualizations here
http://rossb34.github.io/PortfolioAnalyticsPresentation2016/#22 on
slides 2226 and code
https://github.com/rossb34/PortfolioAnalyticsPresentation2016/blob/master/feasible_space.Rto produce the plots.
Ross
On Tue, Mar 21, 2017 at 5:41 AM, Brian G. Peterson < [hidden email]> wrote:
> The process you describe is pretty standard for an investmentcommittee
> driven process.
>
> I'm going to suggest that you don't really want to change the constraints
> that often. For example, box constraints should be as large as your overall
> investment mandate allows to give you the greatest possible room for
> allocations. Sector or Factor constraints likewise should be as minimal as
> possible just to guarantee the degree of diversification described in your
> investment mandate.
>
> The reason I'm suggesting this minimal constraint set is one of the reasons
> we wrote the random portfolio code in the first place. To see what I mean,
> generate a set of unconstrained random portfolios (or e.g. only with a
> fullinvestment constraint). Then generate sets of constrained random
> portfolios, adding your various constraint sets. Plot the different sets on
> the same risk/return scatter plot, using different colors for each set.
> Note how small the feasible space becomes, very quickly.
>
> This shrinkage of the feasible space has some good shrinkage properties...
> moderate shrinkage actually decreases the possible impact of estimation
> error in the various inputs, a little. Large amounts of shrinkage (overly
> restrictive constraints) will do the opposite, and magnify the negative out
> of sample impact of estimation error.
>
> The academic literature mostly focuses on analytical solvers (e.g.
> quadratic, linear, etc) and simple constraint sets. We've cited papers by
> Patrick Burns as well as papers on the simplex models in PortfolioAnalytics,
> but the literature is not vast.
>
> Numerical solvers become important as the feasible space becomes nonsmooth.
> One of the things that can create a nonsmooth feasible space is a complex,
> overlapping constraint set.
>
> The rportfolios package proposed by Frederick Novomestky also seems to be an
> Ronly implementation, at a glance relying on truncated random binomial
> vectors rather than truncated random uniform vectors. I believe it will
> have similar performance characteristics to the Burnsstyle random sample
> portfolios, and it seems to support fewer constraint sets (no overlapping
> sector, group, or factor constraints that I see). In any case, it generates
> matrices of weights that are likely compatible with the PortfolioAnalytics
> random or seed portfolio inputs. So if it works for you, that's great.
>
> You also discuss using rejection after generating the portfolios. This is
> the method used internally by random.portfolios to reject individual weights
> if a constraint is violated. I'll have to evaluate whether the truncdist
> package used by rportfolios could be more efficient than the runif that is
> used by the current code. PortfolioAnalytics also allows portfolios to be
> penalized in the solver, so that more complex cases can be considered, or
> interactions between constraints and objectives.
>
> To answer the question of whether Rcpp will help is somewhat complex. I'm
> confident that some of the nested loops in the generation code will be sped
> up by Rcpp. It is possible that more efficient algorithms are available for
> constructing the weight vectors. A reason that this hasn't been a huge
> priority though is that construction of the random portfolio matrix is
> usually not the time limiter in a large optimization: your objective
> function is. I think it will be possible to improve the efficiency of this
> step, though it is unclear how much of an impact this should have in
> practice to a large and complicated numerically solved portfolio
> optimization problem.
>
> Regards,
>
> Brian
>
>
>
>
> On 03/20/2017 07:06 PM, Kevin Dhingra wrote:
>>
>> Brian,
>>
>> Yes I think that will be a good starting point. My universe would not
>> change a lot (I will be working with 1015 benchmarks at a time and I
>> guess
>> I can generate a reusable set for each independently before running it
>> through my main algorithm). Having said that, I envision the investment
>> mandates/constraints changing quite a lot (both in the cross section and
>> also over time for the same manager). I am hoping there must be a way
>> around it using rejection sampling but have not done enough research to
>> comment on how that solution works for such big dimensions. It will be
>> really helpful if you could point me to any specific resources from
>> academia for the same (Haven't been able to find much about random
>> portfolios myself except Portfolio Analytics and Patrick Burns work on
>> Portfolio Probe). As a side note  Do you think translating it using Rcpp
>> would be time well spent or you think there must be a smarter way to get
>> around it still using R?
>>
>> I really appreciate your help on this thread.
>>
>> Regards,
>> Kshitij Dhingra
>>
>> On Mon, Mar 20, 2017 at 7:21 PM, Brian G. Peterson < [hidden email]>
>> wrote:
>>
>>> For this type of problem, I would probably generate one set of random
>>> portfolios and just reuse that set of feasible portfolios... My usual
>>> rule
>>> is nassets + 12k feasible portfolios. You can get a better number e.g.
>>> from sampling theory, but this should be enough.
>>>
>>> Once you have this weights matrix rp, you only need to regenerate rp if
>>> your universe changes.
>>>
>>> Still interested in a more efficient implementation, of course, or we can
>>> work with you to see if we can find resources to work on it, e.g. from
>>> academia.
>>>
>>> Regards,
>>>
>>> Brian
>>>
>>>
>>> On 03/20/2017 05:28 PM, Kevin Dhingra wrote:
>>>
>>>> Hi Ross,
>>>>
>>>> Sure. Even though I have not profiled the bottlenecks quite in detail as
>>>> of
>>>> yet, i will give you a decent idea of the problem I am working with. I
>>>> can
>>>> have multiple indices with as much as 2000 assets with group, position
>>>> and
>>>> turnover limits (Not sure if i can increase the speed by removing
>>>> constraints and doing rejection sampling later). In order to generate a
>>>> daily possible set for the market in this case, I was playing around
>>>> with
>>>> ~45 thousand permutations. Also I think I will end up using the
>>>> "sample"
>>>> method because of the type of constraints we have and as you already
>>>> have
>>>> mentioned that method is the slowest (takes about 30 times the time
>>>> using
>>>> "simplex" for the same constraints). Adding box and position limit
>>>> constraints are causing it to run a bit slower (but its not a big
>>>> difference). I can always provide a more thorough analysis of the
>>>> potential
>>>> bottlenecks with a lot more detail when I have a chance to start working
>>>> on
>>>> translating it to cpp
>>>>
>>>> Thank you,
>>>>
>>>> On Mon, Mar 20, 2017 at 4:04 PM, Ross Bennett < [hidden email]>
>>>> wrote:
>>>>
>>>> Kevin,
>>>>>
>>>>>
>>>>> Can you give us a sense of the number of assets in the portfolio and
>>>>> the constraints? That will help us understand where the potential
>>>>> bottlenecks are in the random portfolio generation. For example,
>>>>> generating a set of random portfolios for box and weight constraints
>>>>> if relatively fast, but adding group or position limit constraints
>>>>> makes the algorithm more complicated and slower.
>>>>>
>>>>> Thanks,
>>>>> Ross
>>>>>
>>>>>
>>>>> On Mon, Mar 20, 2017 at 2:35 PM, Kevin Dhingra
>>>>> < [hidden email]> wrote:
>>>>>
>>>>>> Brian,
>>>>>>
>>>>>> Thank you for a quick reply. I will soon be working on that problem
>>>>>> and
>>>>>> from what I have played with so far, it is unlikely that for our
>>>>>> example
>>>>>> ~2k portfolios will be enough (really hoping it would) to get a good
>>>>>>
>>>>> sense
>>>>>
>>>>>> of the feasible space and seems like I need to implement an Rcpp
>>>>>> version
>>>>>>
>>>>> of
>>>>>
>>>>>> the random portfolios function. I will be happy to collaborate and
>>>>>> share
>>>>>>
>>>>> my
>>>>>
>>>>>> code once i get a decent handle on it locally for the purposes of our
>>>>>> current project.
>>>>>>
>>>>>> Regards,
>>>>>> Kshitij Dhingra
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 20, 2017 at 3:17 PM, Brian G. Peterson
>>>>>> < [hidden email]
>>>>>>>
>>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> On Mon, 20170320 at 15:09 0400, Kevin Dhingra wrote:
>>>>>>>
>>>>>>>
>>>>>>>> I have been using the random_portfolios function from the
>>>>>>>> `PortfolioAnalytics` package to simulate the range of possibilities
>>>>>>>> for return paths at each step under various portfolio constraints /
>>>>>>>> mandates for evaluating mutual fund managers. As more managers are
>>>>>>>> added to the universe, however, and more simulations are needed, the
>>>>>>>> pure R implementations get pretty heavy and hard to scale. I was
>>>>>>>> wondering if there has been any work out there thus far on
>>>>>>>> implementing any of the three random portfolio generation methods
>>>>>>>> (sample, simplex, and grid search) at a lower level, using something
>>>>>>>> like `Rcpp` to enhance the efficiency of these algorithms?
>>>>>>>>
>>>>>>>
>>>>>>> We've discussed it, but I can't say that it is terribly high on our
>>>>>>> list of priorities.
>>>>>>>
>>>>>>> In most cases, no more than 12k portfolios should be required to get
>>>>>>> a
>>>>>>> fair view of the feasible space given your constraints and
>>>>>>> objectives.
>>>>>>>
>>>>>>> We'd be happy to work with you if you want to craft a patch to use C
>>>>>>> or
>>>>>>> Rcpp for this.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Brian
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> 
>>>>>> Kshitij Dhingra
>>>>>> Applied Academics LLC
>>>>>> Office: +1.917.262.0516
>>>>>> Mobile: +1.206.696.5945
>>>>>> Email: [hidden email]
>>>>>> Website: http://www.AppliedAcademics.com>>>>>>
>>>>>> [[alternative HTML version deleted]]
>>>>>>
>>>>>> _______________________________________________
>>>>>> [hidden email] mailing list
>>>>>> https://stat.ethz.ch/mailman/listinfo/rsigfinance>>>>>>  Subscriberposting only. If you want to post, subscribe first.
>>>>>>  Also note that this is not the rhelp list where general R
>>>>>> questions
>>>>>>
>>>>> should go.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>> 
>>> Brian G. Peterson
>>> http://braverock.com/brian/>>> Ph: 7734594973
>>> IM: bgpbraverock
>>>
>>>
>>> _______________________________________________
>>> [hidden email] mailing list
>>> https://stat.ethz.ch/mailman/listinfo/rsigfinance>>>  Subscriberposting only. If you want to post, subscribe first.
>>>  Also note that this is not the rhelp list where general R questions
>>> should go.
>>>
>>
>>
>>
>
>
> 
> Brian G. Peterson
> http://braverock.com/brian/> Ph: 7734594973
> IM: bgpbraverock
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/rsigfinance>  Subscriberposting only. If you want to post, subscribe first.
>  Also note that this is not the rhelp list where general R questions
> should go.
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


Brian, Ross and Frednovo,
This has been extremely helpful. I think now I have a lot better
understanding of how to think about the problem at hand. Appreciate it.
Regards,
Kshitij Dhingra
On Tue, Mar 21, 2017 at 7:24 AM, Ross Bennett < [hidden email]>
wrote:
> Just wanted to point out an example of what Brian mentioned for
> visualizing the feasible space with different constraint sets.
> Visualizations here
> http://rossb34.github.io/PortfolioAnalyticsPresentation2016/#22 on
> slides 2226 and code
> https://github.com/rossb34/PortfolioAnalyticsPresentation> 2016/blob/master/feasible_space.R
> to produce the plots.
>
> Ross
>
>
> On Tue, Mar 21, 2017 at 5:41 AM, Brian G. Peterson < [hidden email]>
> wrote:
> > The process you describe is pretty standard for an investmentcommittee
> > driven process.
> >
> > I'm going to suggest that you don't really want to change the constraints
> > that often. For example, box constraints should be as large as your
> overall
> > investment mandate allows to give you the greatest possible room for
> > allocations. Sector or Factor constraints likewise should be as minimal
> as
> > possible just to guarantee the degree of diversification described in
> your
> > investment mandate.
> >
> > The reason I'm suggesting this minimal constraint set is one of the
> reasons
> > we wrote the random portfolio code in the first place. To see what I
> mean,
> > generate a set of unconstrained random portfolios (or e.g. only with a
> > fullinvestment constraint). Then generate sets of constrained random
> > portfolios, adding your various constraint sets. Plot the different sets
> on
> > the same risk/return scatter plot, using different colors for each set.
> > Note how small the feasible space becomes, very quickly.
> >
> > This shrinkage of the feasible space has some good shrinkage
> properties...
> > moderate shrinkage actually decreases the possible impact of estimation
> > error in the various inputs, a little. Large amounts of shrinkage
> (overly
> > restrictive constraints) will do the opposite, and magnify the negative
> out
> > of sample impact of estimation error.
> >
> > The academic literature mostly focuses on analytical solvers (e.g.
> > quadratic, linear, etc) and simple constraint sets. We've cited papers
> by
> > Patrick Burns as well as papers on the simplex models in
> PortfolioAnalytics,
> > but the literature is not vast.
> >
> > Numerical solvers become important as the feasible space becomes
> nonsmooth.
> > One of the things that can create a nonsmooth feasible space is a
> complex,
> > overlapping constraint set.
> >
> > The rportfolios package proposed by Frederick Novomestky also seems to
> be an
> > Ronly implementation, at a glance relying on truncated random binomial
> > vectors rather than truncated random uniform vectors. I believe it will
> > have similar performance characteristics to the Burnsstyle random sample
> > portfolios, and it seems to support fewer constraint sets (no overlapping
> > sector, group, or factor constraints that I see). In any case, it
> generates
> > matrices of weights that are likely compatible with the
> PortfolioAnalytics
> > random or seed portfolio inputs. So if it works for you, that's great.
> >
> > You also discuss using rejection after generating the portfolios. This
> is
> > the method used internally by random.portfolios to reject individual
> weights
> > if a constraint is violated. I'll have to evaluate whether the truncdist
> > package used by rportfolios could be more efficient than the runif that
> is
> > used by the current code. PortfolioAnalytics also allows portfolios to
> be
> > penalized in the solver, so that more complex cases can be considered, or
> > interactions between constraints and objectives.
> >
> > To answer the question of whether Rcpp will help is somewhat complex. I'm
> > confident that some of the nested loops in the generation code will be
> sped
> > up by Rcpp. It is possible that more efficient algorithms are available
> for
> > constructing the weight vectors. A reason that this hasn't been a huge
> > priority though is that construction of the random portfolio matrix is
> > usually not the time limiter in a large optimization: your objective
> > function is. I think it will be possible to improve the efficiency of
> this
> > step, though it is unclear how much of an impact this should have in
> > practice to a large and complicated numerically solved portfolio
> > optimization problem.
> >
> > Regards,
> >
> > Brian
> >
> >
> >
> >
> > On 03/20/2017 07:06 PM, Kevin Dhingra wrote:
> >>
> >> Brian,
> >>
> >> Yes I think that will be a good starting point. My universe would not
> >> change a lot (I will be working with 1015 benchmarks at a time and I
> >> guess
> >> I can generate a reusable set for each independently before running it
> >> through my main algorithm). Having said that, I envision the investment
> >> mandates/constraints changing quite a lot (both in the cross section and
> >> also over time for the same manager). I am hoping there must be a way
> >> around it using rejection sampling but have not done enough research to
> >> comment on how that solution works for such big dimensions. It will be
> >> really helpful if you could point me to any specific resources from
> >> academia for the same (Haven't been able to find much about random
> >> portfolios myself except Portfolio Analytics and Patrick Burns work on
> >> Portfolio Probe). As a side note  Do you think translating it using
> Rcpp
> >> would be time well spent or you think there must be a smarter way to get
> >> around it still using R?
> >>
> >> I really appreciate your help on this thread.
> >>
> >> Regards,
> >> Kshitij Dhingra
> >>
> >> On Mon, Mar 20, 2017 at 7:21 PM, Brian G. Peterson < [hidden email]
> >
> >> wrote:
> >>
> >>> For this type of problem, I would probably generate one set of random
> >>> portfolios and just reuse that set of feasible portfolios... My usual
> >>> rule
> >>> is nassets + 12k feasible portfolios. You can get a better number
> e.g.
> >>> from sampling theory, but this should be enough.
> >>>
> >>> Once you have this weights matrix rp, you only need to regenerate rp if
> >>> your universe changes.
> >>>
> >>> Still interested in a more efficient implementation, of course, or we
> can
> >>> work with you to see if we can find resources to work on it, e.g. from
> >>> academia.
> >>>
> >>> Regards,
> >>>
> >>> Brian
> >>>
> >>>
> >>> On 03/20/2017 05:28 PM, Kevin Dhingra wrote:
> >>>
> >>>> Hi Ross,
> >>>>
> >>>> Sure. Even though I have not profiled the bottlenecks quite in detail
> as
> >>>> of
> >>>> yet, i will give you a decent idea of the problem I am working with. I
> >>>> can
> >>>> have multiple indices with as much as 2000 assets with group, position
> >>>> and
> >>>> turnover limits (Not sure if i can increase the speed by removing
> >>>> constraints and doing rejection sampling later). In order to generate
> a
> >>>> daily possible set for the market in this case, I was playing around
> >>>> with
> >>>> ~45 thousand permutations. Also I think I will end up using the
> >>>> "sample"
> >>>> method because of the type of constraints we have and as you already
> >>>> have
> >>>> mentioned that method is the slowest (takes about 30 times the time
> >>>> using
> >>>> "simplex" for the same constraints). Adding box and position limit
> >>>> constraints are causing it to run a bit slower (but its not a big
> >>>> difference). I can always provide a more thorough analysis of the
> >>>> potential
> >>>> bottlenecks with a lot more detail when I have a chance to start
> working
> >>>> on
> >>>> translating it to cpp
> >>>>
> >>>> Thank you,
> >>>>
> >>>> On Mon, Mar 20, 2017 at 4:04 PM, Ross Bennett <
> [hidden email]>
> >>>> wrote:
> >>>>
> >>>> Kevin,
> >>>>>
> >>>>>
> >>>>> Can you give us a sense of the number of assets in the portfolio and
> >>>>> the constraints? That will help us understand where the potential
> >>>>> bottlenecks are in the random portfolio generation. For example,
> >>>>> generating a set of random portfolios for box and weight constraints
> >>>>> if relatively fast, but adding group or position limit constraints
> >>>>> makes the algorithm more complicated and slower.
> >>>>>
> >>>>> Thanks,
> >>>>> Ross
> >>>>>
> >>>>>
> >>>>> On Mon, Mar 20, 2017 at 2:35 PM, Kevin Dhingra
> >>>>> < [hidden email]> wrote:
> >>>>>
> >>>>>> Brian,
> >>>>>>
> >>>>>> Thank you for a quick reply. I will soon be working on that problem
> >>>>>> and
> >>>>>> from what I have played with so far, it is unlikely that for our
> >>>>>> example
> >>>>>> ~2k portfolios will be enough (really hoping it would) to get a good
> >>>>>>
> >>>>> sense
> >>>>>
> >>>>>> of the feasible space and seems like I need to implement an Rcpp
> >>>>>> version
> >>>>>>
> >>>>> of
> >>>>>
> >>>>>> the random portfolios function. I will be happy to collaborate and
> >>>>>> share
> >>>>>>
> >>>>> my
> >>>>>
> >>>>>> code once i get a decent handle on it locally for the purposes of
> our
> >>>>>> current project.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Kshitij Dhingra
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Mar 20, 2017 at 3:17 PM, Brian G. Peterson
> >>>>>> < [hidden email]
> >>>>>>>
> >>>>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>> On Mon, 20170320 at 15:09 0400, Kevin Dhingra wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>> I have been using the random_portfolios function from the
> >>>>>>>> `PortfolioAnalytics` package to simulate the range of
> possibilities
> >>>>>>>> for return paths at each step under various portfolio constraints
> /
> >>>>>>>> mandates for evaluating mutual fund managers. As more managers are
> >>>>>>>> added to the universe, however, and more simulations are needed,
> the
> >>>>>>>> pure R implementations get pretty heavy and hard to scale. I was
> >>>>>>>> wondering if there has been any work out there thus far on
> >>>>>>>> implementing any of the three random portfolio generation methods
> >>>>>>>> (sample, simplex, and grid search) at a lower level, using
> something
> >>>>>>>> like `Rcpp` to enhance the efficiency of these algorithms?
> >>>>>>>>
> >>>>>>>
> >>>>>>> We've discussed it, but I can't say that it is terribly high on our
> >>>>>>> list of priorities.
> >>>>>>>
> >>>>>>> In most cases, no more than 12k portfolios should be required to
> get
> >>>>>>> a
> >>>>>>> fair view of the feasible space given your constraints and
> >>>>>>> objectives.
> >>>>>>>
> >>>>>>> We'd be happy to work with you if you want to craft a patch to use
> C
> >>>>>>> or
> >>>>>>> Rcpp for this.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>>
> >>>>>>> Brian
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> 
> >>>>>> Kshitij Dhingra
> >>>>>> Applied Academics LLC
> >>>>>> Office: +1.917.262.0516
> >>>>>> Mobile: +1.206.696.5945
> >>>>>> Email: [hidden email]
> >>>>>> Website: http://www.AppliedAcademics.com> >>>>>>
> >>>>>> [[alternative HTML version deleted]]
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> [hidden email] mailing list
> >>>>>> https://stat.ethz.ch/mailman/listinfo/rsigfinance> >>>>>>  Subscriberposting only. If you want to post, subscribe first.
> >>>>>>  Also note that this is not the rhelp list where general R
> >>>>>> questions
> >>>>>>
> >>>>> should go.
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>> 
> >>> Brian G. Peterson
> >>> http://braverock.com/brian/> >>> Ph: 7734594973
> >>> IM: bgpbraverock
> >>>
> >>>
> >>> _______________________________________________
> >>> [hidden email] mailing list
> >>> https://stat.ethz.ch/mailman/listinfo/rsigfinance> >>>  Subscriberposting only. If you want to post, subscribe first.
> >>>  Also note that this is not the rhelp list where general R questions
> >>> should go.
> >>>
> >>
> >>
> >>
> >
> >
> > 
> > Brian G. Peterson
> > http://braverock.com/brian/> > Ph: 7734594973
> > IM: bgpbraverock
> >
> > _______________________________________________
> > [hidden email] mailing list
> > https://stat.ethz.ch/mailman/listinfo/rsigfinance> >  Subscriberposting only. If you want to post, subscribe first.
> >  Also note that this is not the rhelp list where general R questions
> > should go.
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/rsigfinance>  Subscriberposting only. If you want to post, subscribe first.
>  Also note that this is not the rhelp list where general R questions
> should go.
>

Kshitij Dhingra
Applied Academics LLC
Office: +1.917.262.0516
Mobile: +1.206.696.5945
Email: [hidden email]
Website: http://www.AppliedAcademics.com [[alternative HTML version deleted]]
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


On Tue, Mar 21, 2017 at 5:41 AM, Brian G. Peterson < [hidden email]> wrote:
>
<snip>
> To answer the question of whether Rcpp will help is somewhat complex. I'm
> confident that some of the nested loops in the generation code will be sped
> up by Rcpp. It is possible that more efficient algorithms are available for
> constructing the weight vectors. A reason that this hasn't been a huge
> priority though is that construction of the random portfolio matrix is
> usually not the time limiter in a large optimization: your objective
> function is. I think it will be possible to improve the efficiency of this
> step, though it is unclear how much of an impact this should have in
> practice to a large and complicated numerically solved portfolio
> optimization problem.
>
Brian has hinted at this, but I want to say it explicitly. Whether
or not moving to compiled code is worth it is mostly an empirical
question. And it's difficult to do more that speculate, unless you
have profiling data. So I would strongly encourage you to profile
your optimization before you change any code. I would be happy to
help review the profiling output.
> Regards,
>
> Brian
>
>
<snip>

Joshua Ulrich  about.me/joshuaulrich
FOSS Trading  www.fosstrading.com
R/Finance 2017  www.rinfinance.com
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


FYI I have found it very useful to use random portfolios to verify that
your optimization is actually doing something. Particular mandates get so
bogged down by constraints (liquidity, box, sector, industry, Barbra risk,
etc) that every answer in the feasible set is basically the same. (i.e. If
you are making money it's on the constraints.)
 Scott
On Tue, Mar 21, 2017 at 13:43 Joshua Ulrich < [hidden email]> wrote:
> On Tue, Mar 21, 2017 at 5:41 AM, Brian G. Peterson < [hidden email]>
> wrote:
> >
> <snip>
> > To answer the question of whether Rcpp will help is somewhat complex. I'm
> > confident that some of the nested loops in the generation code will be
> sped
> > up by Rcpp. It is possible that more efficient algorithms are available
> for
> > constructing the weight vectors. A reason that this hasn't been a huge
> > priority though is that construction of the random portfolio matrix is
> > usually not the time limiter in a large optimization: your objective
> > function is. I think it will be possible to improve the efficiency of
> this
> > step, though it is unclear how much of an impact this should have in
> > practice to a large and complicated numerically solved portfolio
> > optimization problem.
> >
> Brian has hinted at this, but I want to say it explicitly. Whether
> or not moving to compiled code is worth it is mostly an empirical
> question. And it's difficult to do more that speculate, unless you
> have profiling data. So I would strongly encourage you to profile
> your optimization before you change any code. I would be happy to
> help review the profiling output.
>
> > Regards,
> >
> > Brian
> >
> >
> <snip>
> >
> >
> > 
> > Brian G. Peterson
> > http://braverock.com/brian/> > Ph: 7734594973
> > IM: bgpbraverock
> >
> > _______________________________________________
> > [hidden email] mailing list
> > https://stat.ethz.ch/mailman/listinfo/rsigfinance> >  Subscriberposting only. If you want to post, subscribe first.
> >  Also note that this is not the rhelp list where general R questions
> > should go.
>
>
>
> 
> Joshua Ulrich  about.me/joshuaulrich
> FOSS Trading  www.fosstrading.com
> R/Finance 2017  www.rinfinance.com
>
> _______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/rsigfinance>  Subscriberposting only. If you want to post, subscribe first.
>  Also note that this is not the rhelp list where general R questions
> should go.
>

Scott Payseur
[[alternative HTML version deleted]]
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.


I'm coming significantly late to the thread, but I'll throw in a bit of
my view on the required number of random portfolios to generate.
I think it depends a whole lot on what you are doing.
If you are essentially looking for a pvalue, then you'll need a few
hundred at most. You shouldn't really care if the pvalue is 4.3%
versus 4.1%, and if you're looking at tiny pvalues, then algorithmic
limitations and the subtleties of what "random" means in the particular
situation are going to start to have big impacts.
If you are looking at densities, then something like 10,000 portfolios
is not terrible. If you are looking at tails, then you want lots more.
But again, you have lots of model risk in the tails.
Pat
On 20/03/2017 23:21, Brian G. Peterson wrote:
> For this type of problem, I would probably generate one set of random
> portfolios and just reuse that set of feasible portfolios... My usual
> rule is nassets + 12k feasible portfolios. You can get a better
> number e.g. from sampling theory, but this should be enough.
>
> Once you have this weights matrix rp, you only need to regenerate rp if
> your universe changes.
>
> Still interested in a more efficient implementation, of course, or we
> can work with you to see if we can find resources to work on it, e.g.
> from academia.
>
> Regards,
>
> Brian
>
> On 03/20/2017 05:28 PM, Kevin Dhingra wrote:
>> Hi Ross,
>>
>> Sure. Even though I have not profiled the bottlenecks quite in detail
>> as of
>> yet, i will give you a decent idea of the problem I am working with. I
>> can
>> have multiple indices with as much as 2000 assets with group, position
>> and
>> turnover limits (Not sure if i can increase the speed by removing
>> constraints and doing rejection sampling later). In order to generate a
>> daily possible set for the market in this case, I was playing around with
>> ~45 thousand permutations. Also I think I will end up using the "sample"
>> method because of the type of constraints we have and as you already have
>> mentioned that method is the slowest (takes about 30 times the time using
>> "simplex" for the same constraints). Adding box and position limit
>> constraints are causing it to run a bit slower (but its not a big
>> difference). I can always provide a more thorough analysis of the
>> potential
>> bottlenecks with a lot more detail when I have a chance to start
>> working on
>> translating it to cpp
>>
>> Thank you,
>>
>> On Mon, Mar 20, 2017 at 4:04 PM, Ross Bennett < [hidden email]>
>> wrote:
>>
>>> Kevin,
>>>
>>> Can you give us a sense of the number of assets in the portfolio and
>>> the constraints? That will help us understand where the potential
>>> bottlenecks are in the random portfolio generation. For example,
>>> generating a set of random portfolios for box and weight constraints
>>> if relatively fast, but adding group or position limit constraints
>>> makes the algorithm more complicated and slower.
>>>
>>> Thanks,
>>> Ross
>>>
>>>
>>> On Mon, Mar 20, 2017 at 2:35 PM, Kevin Dhingra
>>> < [hidden email]> wrote:
>>>> Brian,
>>>>
>>>> Thank you for a quick reply. I will soon be working on that problem and
>>>> from what I have played with so far, it is unlikely that for our
>>>> example
>>>> ~2k portfolios will be enough (really hoping it would) to get a good
>>> sense
>>>> of the feasible space and seems like I need to implement an Rcpp
>>>> version
>>> of
>>>> the random portfolios function. I will be happy to collaborate and
>>>> share
>>> my
>>>> code once i get a decent handle on it locally for the purposes of our
>>>> current project.
>>>>
>>>> Regards,
>>>> Kshitij Dhingra
>>>>
>>>>
>>>>
>>>> On Mon, Mar 20, 2017 at 3:17 PM, Brian G. Peterson
>>>> < [hidden email]>
>>>> wrote:
>>>>
>>>>> On Mon, 20170320 at 15:09 0400, Kevin Dhingra wrote:
>>>>>> I have been using the random_portfolios function from the
>>>>>> `PortfolioAnalytics` package to simulate the range of possibilities
>>>>>> for return paths at each step under various portfolio constraints /
>>>>>> mandates for evaluating mutual fund managers. As more managers are
>>>>>> added to the universe, however, and more simulations are needed, the
>>>>>> pure R implementations get pretty heavy and hard to scale. I was
>>>>>> wondering if there has been any work out there thus far on
>>>>>> implementing any of the three random portfolio generation methods
>>>>>> (sample, simplex, and grid search) at a lower level, using something
>>>>>> like `Rcpp` to enhance the efficiency of these algorithms?
>>>>>
>>>>> We've discussed it, but I can't say that it is terribly high on our
>>>>> list of priorities.
>>>>>
>>>>> In most cases, no more than 12k portfolios should be required to
>>>>> get a
>>>>> fair view of the feasible space given your constraints and objectives.
>>>>>
>>>>> We'd be happy to work with you if you want to craft a patch to use
>>>>> C or
>>>>> Rcpp for this.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Brian
>>>>>
>>>>
>>>>
>>>>
>>>> 
>>>> Kshitij Dhingra
>>>> Applied Academics LLC
>>>> Office: +1.917.262.0516
>>>> Mobile: +1.206.696.5945
>>>> Email: [hidden email]
>>>> Website: http://www.AppliedAcademics.com>>>>
>>>> [[alternative HTML version deleted]]
>>>>
>>>> _______________________________________________
>>>> [hidden email] mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/rsigfinance>>>>  Subscriberposting only. If you want to post, subscribe first.
>>>>  Also note that this is not the rhelp list where general R questions
>>> should go.
>>>
>>
>>
>>
>
>

Patrick Burns
[hidden email]
http://www.burnsstat.comhttp://www.portfolioprobe.com/blogtwitter: @burnsstat @portfolioprobe
_______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/rsigfinance Subscriberposting only. If you want to post, subscribe first.
 Also note that this is not the rhelp list where general R questions should go.

