Efficient manipulation with list object

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Efficient manipulation with list object

Bogaso
Hi,

I have a list of length 10,000, and each element of that list is a matrix
with 3 columns and 2,000 rows.

Now when I tried to make a Matrix object with that list using
Reduce('rbind', list), my code is taking a considerable amount of time.

Is there any way to implement same above task in more efficient way?

Thanks,

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Efficient manipulation with list object

Eric Berger
Try this. Suppose your list of matrices is in the list locL.

nc <- 3

locL2 <- list()
for ( i in 1:length(locL )
  locL2[[i]] <- as.numeric(t(locL[[i]]))

bigMat <- matrix(unlist(locL3), ncol=nc, byrow=TRUE)

HTH,
Eric


On Sun, Jun 10, 2018 at 6:33 PM, Christofer Bogaso <
[hidden email]> wrote:

> Hi,
>
> I have a list of length 10,000, and each element of that list is a matrix
> with 3 columns and 2,000 rows.
>
> Now when I tried to make a Matrix object with that list using
> Reduce('rbind', list), my code is taking a considerable amount of time.
>
> Is there any way to implement same above task in more efficient way?
>
> Thanks,
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/
> posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Efficient manipulation with list object

Eric Berger
Sorry typos

Try this. Suppose your list of matrices is in the list locL.

nc <- 3

locL2 <- list()
for ( i in 1:length(locL) )
  locL2[[i]] <- as.numeric( t( locL[[i]] ) )

bigMat <- matrix(unlist(locL2), ncol=nc, byrow=TRUE)

HTH,
Eric


On Sun, Jun 10, 2018 at 7:10 PM, Eric Berger <[hidden email]> wrote:

> Try this. Suppose your list of matrices is in the list locL.
>
> nc <- 3
>
> locL2 <- list()
> for ( i in 1:length(locL )
>   locL2[[i]] <- as.numeric(t(locL[[i]]))
>
> bigMat <- matrix(unlist(locL3), ncol=nc, byrow=TRUE)
>
> HTH,
> Eric
>
>
> On Sun, Jun 10, 2018 at 6:33 PM, Christofer Bogaso <
> [hidden email]> wrote:
>
>> Hi,
>>
>> I have a list of length 10,000, and each element of that list is a matrix
>> with 3 columns and 2,000 rows.
>>
>> Now when I tried to make a Matrix object with that list using
>> Reduce('rbind', list), my code is taking a considerable amount of time.
>>
>> Is there any way to implement same above task in more efficient way?
>>
>> Thanks,
>>
>>         [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posti
>> ng-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Efficient manipulation with list object

Rui Barradas
In reply to this post by Bogaso
Hello,
Instead of Reduce try do.call.
do.call ('rbind', list)
But with such a long list it will still take time.
Hope this helps,
Rui BarradasĀ 


Enviado a partir do meu smartphone Samsung Galaxy.-------- Mensagem original --------De: Christofer Bogaso <[hidden email]> Data: 10/06/2018  16:33  (GMT+00:00) Para: r-help <[hidden email]> Assunto: [R] Efficient manipulation with list object
Hi,

I have a list of length 10,000, and each element of that list is a matrix
with 3 columns and 2,000 rows.

Now when I tried to make a Matrix object with that list using
Reduce('rbind', list), my code is taking a considerable amount of time.

Is there any way to implement same above task in more efficient way?

Thanks,

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Efficient manipulation with list object

Bogaso
Using do.call() reduces my calculation time significantly.

On Sun, Jun 10, 2018 at 10:45 PM ruipbarradas <[hidden email]> wrote:

> Hello,
>
> Instead of Reduce try do.call.
>
> do.call ('rbind', list)
>
> But with such a long list it will still take time.
>
> Hope this helps,
>
> Rui Barradas
>
>
>
> Enviado a partir do meu smartphone Samsung Galaxy.
> -------- Mensagem original --------
> De: Christofer Bogaso <[hidden email]>
> Data: 10/06/2018 16:33 (GMT+00:00)
> Para: r-help <[hidden email]>
> Assunto: [R] Efficient manipulation with list object
>
> Hi,
>
> I have a list of length 10,000, and each element of that list is a matrix
> with 3 columns and 2,000 rows.
>
> Now when I tried to make a Matrix object with that list using
> Reduce('rbind', list), my code is taking a considerable amount of time.
>
> Is there any way to implement same above task in more efficient way?
>
> Thanks,
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Efficient manipulation with list object

Benjamin Christoffersen
You may be able to speed it up further by using `data.table`'s
`rbindlist` or a similar function as shown here
https://stackoverflow.com/a/49772719/5861244.

2018-06-10 21:20 GMT+02:00 Christofer Bogaso <[hidden email]>:

> Using do.call() reduces my calculation time significantly.
>
> On Sun, Jun 10, 2018 at 10:45 PM ruipbarradas <[hidden email]> wrote:
>
>> Hello,
>>
>> Instead of Reduce try do.call.
>>
>> do.call ('rbind', list)
>>
>> But with such a long list it will still take time.
>>
>> Hope this helps,
>>
>> Rui Barradas
>>
>>
>>
>> Enviado a partir do meu smartphone Samsung Galaxy.
>> -------- Mensagem original --------
>> De: Christofer Bogaso <[hidden email]>
>> Data: 10/06/2018 16:33 (GMT+00:00)
>> Para: r-help <[hidden email]>
>> Assunto: [R] Efficient manipulation with list object
>>
>> Hi,
>>
>> I have a list of length 10,000, and each element of that list is a matrix
>> with 3 columns and 2,000 rows.
>>
>> Now when I tried to make a Matrix object with that list using
>> Reduce('rbind', list), my code is taking a considerable amount of time.
>>
>> Is there any way to implement same above task in more efficient way?
>>
>> Thanks,
>>
>> [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Efficient manipulation with list object

Jeff Newmiller
The question was about matrices, not data frames or data tables. While faster than Reduce, the conversions still make it over twice as slow as Rui's answer.

On June 10, 2018 12:18:39 PM HST, Benjamin Christoffersen <[hidden email]> wrote:

>You may be able to speed it up further by using `data.table`'s
>`rbindlist` or a similar function as shown here
>https://stackoverflow.com/a/49772719/5861244.
>
>2018-06-10 21:20 GMT+02:00 Christofer Bogaso
><[hidden email]>:
>> Using do.call() reduces my calculation time significantly.
>>
>> On Sun, Jun 10, 2018 at 10:45 PM ruipbarradas <[hidden email]>
>wrote:
>>
>>> Hello,
>>>
>>> Instead of Reduce try do.call.
>>>
>>> do.call ('rbind', list)
>>>
>>> But with such a long list it will still take time.
>>>
>>> Hope this helps,
>>>
>>> Rui Barradas
>>>
>>>
>>>
>>> Enviado a partir do meu smartphone Samsung Galaxy.
>>> -------- Mensagem original --------
>>> De: Christofer Bogaso <[hidden email]>
>>> Data: 10/06/2018 16:33 (GMT+00:00)
>>> Para: r-help <[hidden email]>
>>> Assunto: [R] Efficient manipulation with list object
>>>
>>> Hi,
>>>
>>> I have a list of length 10,000, and each element of that list is a
>matrix
>>> with 3 columns and 2,000 rows.
>>>
>>> Now when I tried to make a Matrix object with that list using
>>> Reduce('rbind', list), my code is taking a considerable amount of
>time.
>>>
>>> Is there any way to implement same above task in more efficient way?
>>>
>>> Thanks,
>>>
>>> [[alternative HTML version deleted]]
>>>
>>> ______________________________________________
>>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>
>>         [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
>______________________________________________
>[hidden email] mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.

--
Sent from my phone. Please excuse my brevity.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.