R Linux Memory Limit error for calculating dissimilarity matrix with Gower distance

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

R Linux Memory Limit error for calculating dissimilarity matrix with Gower distance

Aerenbkts bkts
Hi all,


I am trying to cluster data file with 50K rows and 10 columns with R. Since
data contains mixed data types (nominal, continuous etc.) I am using gower
distance to create a dissimilarity matrix. But while I was creating the
matrix, I got Error: cannot allocate vector of size 4.1 Gb error. I am
using 64 bit 16 GB RAM.

1-My question is how to use all my memory for R (memory.limit does not work
for linux)?
2-Can I chunk dissimilarty matrix while I am creting it with ;

gower_dist <- daisy(i_cluster_ready,metric = "gower")

3-Can daisy function directly write to ROM to avoid error of RAM usace?


I have tried ff and bigmemory packages. Bigmemory does not work with
mixed data. and I could not figure ff function out.


Thanks for your help


Kind regards

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.