Quantcast

Re: getting faster results

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: getting faster results

Elizabeth Lawson-2
I am running algorithms for Bayesian hierarchical models with 20 or more variables or writing alogrithms that solve for partially linear models using wavlelet estimation.  Soemtimes just creating the predicted vaules from the Bayesian analysis takes over a day to run!
  I am just wondering if R runs faster on any particular systems or if it is about the same regardless.

jim holtman <[hidden email]> wrote:
  Need to know what you are trying to do with the data.  Depending on what you want to do and how you do it will determine the speed.  I can take a dataframe with 1,000,000 rows of transaction data and compute the average and plot a histogram of the response time in less that a couple of seconds.  Is that fast enough?

  On 12/13/05, Elizabeth Lawson <[hidden email]> wrote:   Hey,

Can anyone answer this question.  I am working with really large datasets and most of the programs I have been running take quite some time.

I heard that R may be faster in Unix.  I sthis true and if so can anyone reccomend which system and requirements may allow things to go faster for?

Thanks!!

Elizabeth Lawson


---------------------------------


       [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html




--
Jim Holtman
Cincinnati, OH
+1 513 247 0281

What the problem you are trying to solve?  


                       
---------------------------------


        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: getting faster results

Bert Gunter
What is almost certainly critical here are the algorithms you use (to do the
integrations in the Bayesian models) and perhaps a fast BLAS.

-- Bert Gunter
Genentech Non-Clinical Statistics
South San Francisco, CA
 
"The business of the statistician is to catalyze the scientific learning
process."  - George E. P. Box
 
 

> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of
> Elizabeth Lawson
> Sent: Wednesday, December 14, 2005 9:53 AM
> To: jim holtman
> Cc: [hidden email]
> Subject: Re: [R] getting faster results
>
> I am running algorithms for Bayesian hierarchical models with
> 20 or more variables or writing alogrithms that solve for
> partially linear models using wavlelet estimation.  Soemtimes
> just creating the predicted vaules from the Bayesian analysis
> takes over a day to run!
>   I am just wondering if R runs faster on any particular
> systems or if it is about the same regardless.
>
> jim holtman <[hidden email]> wrote:
>   Need to know what you are trying to do with the data.  
> Depending on what you want to do and how you do it will
> determine the speed.  I can take a dataframe with 1,000,000
> rows of transaction data and compute the average and plot a
> histogram of the response time in less that a couple of
> seconds.  Is that fast enough?
>
>   On 12/13/05, Elizabeth Lawson <[hidden email]> wrote:   Hey,
>
> Can anyone answer this question.  I am working with really
> large datasets and most of the programs I have been running
> take quite some time.
>
> I heard that R may be faster in Unix.  I sthis true and if so
> can anyone reccomend which system and requirements may allow
> things to go faster for?
>
> Thanks!!
>
> Elizabeth Lawson
>
>
> ---------------------------------
>
>
>        [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
>
>
>
>
> --
> Jim Holtman
> Cincinnati, OH
> +1 513 247 0281
>
> What the problem you are trying to solve?  
>
>
>
> ---------------------------------
>
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: getting faster results

Liaw, Andy
In reply to this post by Elizabeth Lawson-2
It can depend on a whole list of things why the computations were taking so
long, and just saying what methods you are using to do the computation is
far from sufficient, as there are efficient ways and inefficient ways to do
any computations, and we don't know how you did any of them.  With the
little information you gave, you cannot honestly expect much help.  All
anyone can give you is generic advise.  Mine is as follows:\

- R has profiling facility that tells you what part(s) of the computation is
taking the most time.  Use it.  (See ?Rprof.)

- If you see (or hear) lots of disk activity, but fairly low CPU usage, you
might be short in physical memory, and R is using virtual memory, which will
be _very_ slow.

- Given the same hardware, there's little difference in performance between
different OSes, although Linux seems to have an edge.  Compilers used can
also make a difference.

- Hardware can make a large difference.  On some analyses I did, running the
same code on an Opteron box running 64-bit Linux (SLES) was over six times
faster than on my Pentium M laptop.

Andy

From: Elizabeth Lawson

>
> I am running algorithms for Bayesian hierarchical models with
> 20 or more variables or writing alogrithms that solve for
> partially linear models using wavlelet estimation.  Soemtimes
> just creating the predicted vaules from the Bayesian analysis
> takes over a day to run!
>   I am just wondering if R runs faster on any particular
> systems or if it is about the same regardless.
>
> jim holtman <[hidden email]> wrote:
>   Need to know what you are trying to do with the data.  
> Depending on what you want to do and how you do it will
> determine the speed.  I can take a dataframe with 1,000,000
> rows of transaction data and compute the average and plot a
> histogram of the response time in less that a couple of
> seconds.  Is that fast enough?
>
>   On 12/13/05, Elizabeth Lawson <[hidden email]> wrote:   Hey,
>
> Can anyone answer this question.  I am working with really
> large datasets and most of the programs I have been running
> take quite some time.
>
> I heard that R may be faster in Unix.  I sthis true and if so
> can anyone reccomend which system and requirements may allow
> things to go faster for?
>
> Thanks!!
>
> Elizabeth Lawson
>
>
> ---------------------------------
>
>
>        [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
>
>
>
>
> --
> Jim
> Holtman
> Cincinnati, OH
> +1 513 247 0281
>
> What the problem you are trying to solve?  
>
>
>
> ---------------------------------
>
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
> http://www.R-project.org/posting-guide.html
>
>

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: getting faster results

Elizabeth Lawson-2
In reply to this post by Elizabeth Lawson-2
I am not sure...
  But my employer would like to get me a new machine and considering that R is the program I use most often, he was wondering what type of machine would be the  best to get form me.

jim holtman <[hidden email]> wrote:
  If you job is CPU intensive (it should be unless it is reading in data all the time, or does not have enough memory so it is paging).  Look at the amount of time you are paging.  Otherwise, it is probably using only 1/4 of the available processors if you system has 4 CPUs.  Do you know how many it has?

  On 12/14/05, Elizabeth Lawson <[hidden email]> wrote:     I am running algorithms for Bayesian hierarchical models with 20 or more variables or writing alogrithms that solve for partially linear models using wavlelet estimation.  Soemtimes just creating the predicted vaules from the Bayesian analysis takes over a day to run!
  I am just wondering if R runs faster on any particular systems or if it is about the same regardless.

jim holtman < [hidden email]> wrote:
  Need to know what you are trying to do with the data.  Depending on what you want to do and how you do it will determine the speed.  I can take a dataframe with 1,000,000 rows of transaction data and compute the average and plot a histogram of the response time in less that a couple of seconds.  Is that fast enough?

  On 12/13/05, Elizabeth Lawson <[hidden email] > wrote:   Hey,

Can anyone answer this question.  I am working with really large datasets and most of the programs I have been running take quite some time.

I heard that R may be faster in Unix.  I sthis true and if so can anyone reccomend which system and requirements may allow things to go faster for?

Thanks!!

Elizabeth Lawson


---------------------------------


       [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html




--
Jim Holtman
Cincinnati, OH
+1 513 247 0281

What the problem you are trying to solve?  
 
   
---------------------------------


 




--
Jim Holtman
Cincinnati, OH
+1 513 247 0281

What the problem you are trying to solve?  


                       
---------------------------------


        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html
Loading...