populating a large matrix

[Nick Matzke]

Hi all,

This question is slightly weird.  I am trying to populate a
matrix with equations.  The matrix represents transition
probabilities between states.  A simple example is:

        state1 state2 state3
state1 s d d*d
state2 e s d*e
state3 e*e e*d s

The parameters s, d, and e need to be optimized with an
iterative algorithm.  This means I have to modify, say, d,
and then recalculate the transition probabilities for each cell.

Currently, I do this by making a matrix with the equations
in character format, setting s, e, and d to values, and then
running each cell through parse(eval(text=celltxt)). As follows:

#################################
# Test code:
# Make the text matrix
txtmat = matrix(c("s", "d", "d*d", "e", "s", "d*e", "e*e",
"e*d", "e*d"), nrow=3, byrow=TRUE)

s=0.7
d=0.2
e=0.1

doit <- function(celltxt)
        {
        cellval = eval(parse(text=celltxt))
        return(cellval)
        }

# Calculate the matrix with numerical values
matrix_vals = sapply(X=txtmat, FUN=doit)
valmat = matrix(matrix_vals, nrow=3, byrow=TRUE)
valmat
# End test code
#################################


...however, this seems to get slow for large matrices.
Since I have to optimize all the parameters I need something
that updates the matrix quickly when s, d, or e is changed.
  Perhaps this is a job for pointers in C++ or something,
but I figure there must be a better way in R.

Can anyone think of something more sophisticated than my
current method?

Thanks in advance for any and all help!!

Cheers,
Nick




--
====================================================
Nicholas J. Matzke
Ph.D. Candidate, Graduate Student Researcher

Huelsenbeck Lab
Center for Theoretical Evolutionary Genomics
4151 VLSB (Valley Life Sciences Building)
Department of Integrative Biology
University of California, Berkeley

Graduate Student Instructor, IB200B
Principles of Phylogenetics: Ecology and Evolution
http://ib.berkeley.edu/courses/ib200b/
http://phylo.wikidot.com/


Lab websites:
http://ib.berkeley.edu/people/lab_detail.php?lab=54
http://fisher.berkeley.edu/cteg/hlab.html
Dept. personal page:
http://ib.berkeley.edu/people/students/person_detail.php?person=370
Lab personal page:
http://fisher.berkeley.edu/cteg/members/matzke.html
Lab phone: [phone]
Dept. fax: [phone]

Cell phone: [phone]
Email: [hidden email]

Mailing address:
Department of Integrative Biology
1005 Valley Life Sciences Building #3140
Berkeley, CA 94720-3140

-----------------------------------------------------
"[W]hen people thought the earth was flat, they were wrong.
When people thought the earth was spherical, they were
wrong. But if you think that thinking the earth is spherical
is just as wrong as thinking the earth is flat, then your
view is wronger than both of them put together."

Isaac Asimov (1989). "The Relativity of Wrong." The
Skeptical Inquirer, 14(1), 35-44. Fall 1989.
http://chem.tufts.edu/AnswersInScience/RelativityofWrong.htm

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[Joshua Wiley-2]

Hi Nick,

Off the cuff:

doit2 <- function(s, d, e) {
  matrix(c(s, d, d*d, e, s, d*e, e*e, e*d, s), 3, byrow=FALSE)
}

doit2(s, d, e)

seems like it should offer a substantial speed up.  Unless the matrix
can be populated with arbitrary formulae, I do not see why it should
be a text matrix and then evaluated.  If you have not already, I would
also encourage you to check out OpenMx.  It is a rather flexible
matrix optimizer, and you may be able to get it to do what you want.

http://openmx.psyc.virginia.edu/docs/OpenMx/latest/Likelihood_Matrix.html

Cheers,

Josh

On Tue, Jun 19, 2012 at 8:10 PM, Nick Matzke <[hidden email]> wrote:

> Hi all,
>
> This question is slightly weird.  I am trying to populate a matrix with
> equations.  The matrix represents transition probabilities between states.
>  A simple example is:
>
>        state1  state2  state3
> state1  s       d       d*d
> state2  e       s       d*e
> state3  e*e     e*d     s
>
> The parameters s, d, and e need to be optimized with an iterative algorithm.
>  This means I have to modify, say, d, and then recalculate the transition
> probabilities for each cell.
>
> Currently, I do this by making a matrix with the equations in character
> format, setting s, e, and d to values, and then running each cell through
> parse(eval(text=celltxt)). As follows:
>
> #################################
> # Test code:
> # Make the text matrix
> txtmat = matrix(c("s", "d", "d*d", "e", "s", "d*e", "e*e", "e*d", "e*d"),
> nrow=3, byrow=TRUE)
>
> s=0.7
> d=0.2
> e=0.1
>
> doit <- function(celltxt)
>        {
>        cellval = eval(parse(text=celltxt))
>        return(cellval)
>        }
>
> # Calculate the matrix with numerical values
> matrix_vals = sapply(X=txtmat, FUN=doit)
> valmat = matrix(matrix_vals, nrow=3, byrow=TRUE)
> valmat
> # End test code
> #################################
>
>
> ...however, this seems to get slow for large matrices. Since I have to
> optimize all the parameters I need something that updates the matrix quickly
> when s, d, or e is changed.  Perhaps this is a job for pointers in C++ or
> something, but I figure there must be a better way in R.
>
> Can anyone think of something more sophisticated than my current method?
>
> Thanks in advance for any and all help!!
>
> Cheers,
> Nick
>
>
>
>
> --
> ====================================================
> Nicholas J. Matzke
> Ph.D. Candidate, Graduate Student Researcher
>
> Huelsenbeck Lab
> Center for Theoretical Evolutionary Genomics
> 4151 VLSB (Valley Life Sciences Building)
> Department of Integrative Biology
> University of California, Berkeley
>
> Graduate Student Instructor, IB200B
> Principles of Phylogenetics: Ecology and Evolution
> http://ib.berkeley.edu/courses/ib200b/
> http://phylo.wikidot.com/
>
>
> Lab websites:
> http://ib.berkeley.edu/people/lab_detail.php?lab=54
> http://fisher.berkeley.edu/cteg/hlab.html
> Dept. personal page:
> http://ib.berkeley.edu/people/students/person_detail.php?person=370
> Lab personal page: http://fisher.berkeley.edu/cteg/members/matzke.html
> Lab phone: [phone]
> Dept. fax: [phone]
>
> Cell phone: [phone]
> Email: [hidden email]
>
> Mailing address:
> Department of Integrative Biology
> 1005 Valley Life Sciences Building #3140
> Berkeley, CA 94720-3140
>
> -----------------------------------------------------
> "[W]hen people thought the earth was flat, they were wrong. When people
> thought the earth was spherical, they were wrong. But if you think that
> thinking the earth is spherical is just as wrong as thinking the earth is
> flat, then your view is wronger than both of them put together."
>
> Isaac Asimov (1989). "The Relativity of Wrong." The Skeptical Inquirer,
> 14(1), 35-44. Fall 1989.
> http://chem.tufts.edu/AnswersInScience/RelativityofWrong.htm
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

--
Joshua Wiley
Ph.D. Student, Health Psychology
Programmer Analyst II, Statistical Consulting Group
University of California, Los Angeles
https://joshuawiley.com/

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[Oliver Ruebenacker]

     Hello,

  For an optimization strategy, we need to know more: do you have many
variables? do you have many different formulae? do you re-calculate
after changing only one variable?

     Take care
     Oliver

On Tue, Jun 19, 2012 at 11:10 PM, Nick Matzke <[hidden email]> wrote:

> Hi all,
>
> This question is slightly weird.  I am trying to populate a matrix with
> equations.  The matrix represents transition probabilities between states.
>  A simple example is:
>
>        state1  state2  state3
> state1  s       d       d*d
> state2  e       s       d*e
> state3  e*e     e*d     s
>
> The parameters s, d, and e need to be optimized with an iterative algorithm.
>  This means I have to modify, say, d, and then recalculate the transition
> probabilities for each cell.
>
> Currently, I do this by making a matrix with the equations in character
> format, setting s, e, and d to values, and then running each cell through
> parse(eval(text=celltxt)). As follows:
>
> #################################
> # Test code:
> # Make the text matrix
> txtmat = matrix(c("s", "d", "d*d", "e", "s", "d*e", "e*e", "e*d", "e*d"),
> nrow=3, byrow=TRUE)
>
> s=0.7
> d=0.2
> e=0.1
>
> doit <- function(celltxt)
>        {
>        cellval = eval(parse(text=celltxt))
>        return(cellval)
>        }
>
> # Calculate the matrix with numerical values
> matrix_vals = sapply(X=txtmat, FUN=doit)
> valmat = matrix(matrix_vals, nrow=3, byrow=TRUE)
> valmat
> # End test code
> #################################
>
>
> ...however, this seems to get slow for large matrices. Since I have to
> optimize all the parameters I need something that updates the matrix quickly
> when s, d, or e is changed.  Perhaps this is a job for pointers in C++ or
> something, but I figure there must be a better way in R.
>
> Can anyone think of something more sophisticated than my current method?
>
> Thanks in advance for any and all help!!
>
> Cheers,
> Nick
>
>
>
>
> --
> ====================================================
> Nicholas J. Matzke
> Ph.D. Candidate, Graduate Student Researcher
>
> Huelsenbeck Lab
> Center for Theoretical Evolutionary Genomics
> 4151 VLSB (Valley Life Sciences Building)
> Department of Integrative Biology
> University of California, Berkeley
>
> Graduate Student Instructor, IB200B
> Principles of Phylogenetics: Ecology and Evolution
> http://ib.berkeley.edu/courses/ib200b/
> http://phylo.wikidot.com/
>
>
> Lab websites:
> http://ib.berkeley.edu/people/lab_detail.php?lab=54
> http://fisher.berkeley.edu/cteg/hlab.html
> Dept. personal page:
> http://ib.berkeley.edu/people/students/person_detail.php?person=370
> Lab personal page: http://fisher.berkeley.edu/cteg/members/matzke.html
> Lab phone: [phone]
> Dept. fax: [phone]
>
> Cell phone: [phone]
> Email: [hidden email]
>
> Mailing address:
> Department of Integrative Biology
> 1005 Valley Life Sciences Building #3140
> Berkeley, CA 94720-3140
>
> -----------------------------------------------------
> "[W]hen people thought the earth was flat, they were wrong. When people
> thought the earth was spherical, they were wrong. But if you think that
> thinking the earth is spherical is just as wrong as thinking the earth is
> flat, then your view is wronger than both of them put together."
>
> Isaac Asimov (1989). "The Relativity of Wrong." The Skeptical Inquirer,
> 14(1), 35-44. Fall 1989.
> http://chem.tufts.edu/AnswersInScience/RelativityofWrong.htm
>
> ______________________________________________
> [hidden email] mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

--
Oliver Ruebenacker, Bioinformatics and Network Analysis Consultant
President and Founder of Knowomics
(http://www.knowomics.com/wiki/Oliver_Ruebenacker)
Consultant at Predictive Medicine
(http://predmed.com/people/oliverruebenacker.html)
SBPAX: Turning Bio Knowledge into Math Models (http://www.sbpax.org)

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[Nick Matzke]

Hi! Thanks for the replies, comments below:


On 6/20/12 3:26 AM, Oliver Ruebenacker wrote:
>       Hello,
>
>    For an optimization strategy, we need to know more: do you have many
> variables? do you have many different formulae? do you re-calculate
> after changing only one variable?
>
>       Take care
>       Oliver


The probability of each transition (each cell in the matrix)
will be a function of 1-5 variables, depending on the
complexity of the model.  And actually I am going to use a
Bayesian MCMC approach, changing variables one at a time and
accepting/rejecting the move using Metropolis-Hastings.

(However, I am interested in complementing this with an ML
approach, if you have any advice.)

I've got some libraries working for exponentiating large
matrices, so the main issue is just re-calculating the
relative rates for each cell when a variable changes.



This might work as a function that is called once instead of
for every cell, I will explore this:


On 6/19/12 8:32 PM, Joshua Wiley wrote:
 > Hi Nick,
 >
 > Off the cuff:
 >
 > doit2<- function(s, d, e) {
 >    matrix(c(s, d, d*d, e, s, d*e, e*e, e*d, s), 3,
byrow=FALSE)
 > }
 >
 > doit2(s, d, e)
 >
 > seems like it should offer a substantial speed up.
Unless the matrix
 > can be populated with arbitrary formulae, I do not see
why it should
 > be a text matrix and then evaluated.  If you have not
already, I would
 > also encourage you to check out OpenMx.  It is a rather
flexible
 > matrix optimizer, and you may be able to get it to do
what you want.
 >
 >
http://openmx.psyc.virginia.edu/docs/OpenMx/latest/Likelihood_Matrix.html
 >
 > Cheers,
 >
 > Josh

>>
>>
>>
>>
>> --
>> ====================================================
>> Nicholas J. Matzke
>> Ph.D. Candidate, Graduate Student Researcher
>>
>> Huelsenbeck Lab
>> Center for Theoretical Evolutionary Genomics
>> 4151 VLSB (Valley Life Sciences Building)
>> Department of Integrative Biology
>> University of California, Berkeley
>>
>> Graduate Student Instructor, IB200B
>> Principles of Phylogenetics: Ecology and Evolution
>> http://ib.berkeley.edu/courses/ib200b/
>> http://phylo.wikidot.com/
>>
>>
>> Lab websites:
>> http://ib.berkeley.edu/people/lab_detail.php?lab=54
>> http://fisher.berkeley.edu/cteg/hlab.html
>> Dept. personal page:
>> http://ib.berkeley.edu/people/students/person_detail.php?person=370
>> Lab personal page: http://fisher.berkeley.edu/cteg/members/matzke.html
>> Lab phone: [phone]
>> Dept. fax: [phone]
>>
>> Cell phone: [phone]
>> Email: [hidden email]
>>
>> Mailing address:
>> Department of Integrative Biology
>> 1005 Valley Life Sciences Building #3140
>> Berkeley, CA 94720-3140
>>
>> -----------------------------------------------------
>> "[W]hen people thought the earth was flat, they were wrong. When people
>> thought the earth was spherical, they were wrong. But if you think that
>> thinking the earth is spherical is just as wrong as thinking the earth is
>> flat, then your view is wronger than both of them put together."
>>
>> Isaac Asimov (1989). "The Relativity of Wrong." The Skeptical Inquirer,
>> 14(1), 35-44. Fall 1989.
>> http://chem.tufts.edu/AnswersInScience/RelativityofWrong.htm
>>
>> ______________________________________________
>> [hidden email] mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
>
>

--
====================================================
Nicholas J. Matzke
Ph.D. Candidate, Graduate Student Researcher

Huelsenbeck Lab
Center for Theoretical Evolutionary Genomics
4151 VLSB (Valley Life Sciences Building)
Department of Integrative Biology
University of California, Berkeley

Graduate Student Instructor, IB200B
Principles of Phylogenetics: Ecology and Evolution
http://ib.berkeley.edu/courses/ib200b/
http://phylo.wikidot.com/


Lab websites:
http://ib.berkeley.edu/people/lab_detail.php?lab=54
http://fisher.berkeley.edu/cteg/hlab.html
Dept. personal page:
http://ib.berkeley.edu/people/students/person_detail.php?person=370
Lab personal page:
http://fisher.berkeley.edu/cteg/members/matzke.html
Lab phone: [phone]
Dept. fax: [phone]

Cell phone: [phone]
Email: [hidden email]

Mailing address:
Department of Integrative Biology
1005 Valley Life Sciences Building #3140
Berkeley, CA 94720-3140

-----------------------------------------------------
"[W]hen people thought the earth was flat, they were wrong.
When people thought the earth was spherical, they were
wrong. But if you think that thinking the earth is spherical
is just as wrong as thinking the earth is flat, then your
view is wronger than both of them put together."

Isaac Asimov (1989). "The Relativity of Wrong." The
Skeptical Inquirer, 14(1), 35-44. Fall 1989.
http://chem.tufts.edu/AnswersInScience/RelativityofWrong.htm

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.