Data With Ordinal Responses: Calculate ICC & Assessing Model Fit

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Data With Ordinal Responses: Calculate ICC & Assessing Model Fit

Sidoti, Salvatore A.
To begin with, I'm not a fan of cross-posting. However, I posted my question on Stack Exchange more than two weeks ago, but I have yet to receive a sufficient answer:

https://stats.stackexchange.com/questions/479600/data-with-ordinal-responses-calculate-icc-assessing-model-fit
 
Here's what I've learned since then (hopefully):
 
1) ICC of a CLMM:
Computed like this:
(variance of the random effect) / (variance of the random effect + 1)
If this is correct, I would love to see a reference/citation for it.
 
2) 95% Confidence Interval for the ICC from a CLMM Model
To my current understanding, a confidence interval for an ICC is only obtainable via simulation. I've conducted simulations with GLMM model objects ('lme4' package) and the bootMer() function. Unfortunately, bootMer() will not accept a CLMM model ('ordinal' package).
 
3) Model Fit of a CLMM
Assuming that the model converges without incident, the model summary includes a condition number of the Hessian ('cond.H'). This value should be below 10^4 for a "good fit". This is straightforward enough. However, I am not as sure about the value for 'max.grad', which needs to be "well below 1". The question is, to what magnitude should max.grad < 1 for a decent model fit? My reference is linked below (Christensen, 2019), but it does not elaborate further on this point:
 
https://documentcloud.adobe.com/link/track?uri=urn:aaid:scds:US:b6a61fe2-b851-49ce-b8b1-cd760d290636
 
3) Effect Size of a CLMM
The random variable's effect is determined by a comparison between the full model to a model with only the fixed effects via the anova() function. I found this information on the 'rcompanion' package website:
 
https://rcompanion.org/handbook/G_12.html
 
The output of this particular anova() will include a value named 'LR.stat', the likelihood ratio statistic. The LR.stat is twice the difference of each log-likelihood (absolute value) of the respective models. Is LR.stat the mixed-model version of an "effect size"? If so, how does one determine if the effect is small, large, in-between, etc?

Cheers,
Sal

Salvatore A. Sidoti
PhD Candidate
Behavioral Ecology
The Ohio State University

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Data With Ordinal Responses: Calculate ICC & Assessing Model Fit

Bert Gunter-2
I believe you should post on r-sig-mixed-models, not here. You are more
likely to find the interest and expertise you seek there.

Bert Gunter

"The trouble with having an open mind is that people keep coming along and
sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )


On Mon, Aug 17, 2020 at 3:28 AM Sidoti, Salvatore A. <
[hidden email]> wrote:

> To begin with, I'm not a fan of cross-posting. However, I posted my
> question on Stack Exchange more than two weeks ago, but I have yet to
> receive a sufficient answer:
>
>
> https://stats.stackexchange.com/questions/479600/data-with-ordinal-responses-calculate-icc-assessing-model-fit
>
> Here's what I've learned since then (hopefully):
>
> 1) ICC of a CLMM:
> Computed like this:
> (variance of the random effect) / (variance of the random effect + 1)
> If this is correct, I would love to see a reference/citation for it.
>
> 2) 95% Confidence Interval for the ICC from a CLMM Model
> To my current understanding, a confidence interval for an ICC is only
> obtainable via simulation. I've conducted simulations with GLMM model
> objects ('lme4' package) and the bootMer() function. Unfortunately,
> bootMer() will not accept a CLMM model ('ordinal' package).
>
> 3) Model Fit of a CLMM
> Assuming that the model converges without incident, the model summary
> includes a condition number of the Hessian ('cond.H'). This value should be
> below 10^4 for a "good fit". This is straightforward enough. However, I am
> not as sure about the value for 'max.grad', which needs to be "well below
> 1". The question is, to what magnitude should max.grad < 1 for a decent
> model fit? My reference is linked below (Christensen, 2019), but it does
> not elaborate further on this point:
>
>
> https://documentcloud.adobe.com/link/track?uri=urn:aaid:scds:US:b6a61fe2-b851-49ce-b8b1-cd760d290636
>
> 3) Effect Size of a CLMM
> The random variable's effect is determined by a comparison between the
> full model to a model with only the fixed effects via the anova() function.
> I found this information on the 'rcompanion' package website:
>
> https://rcompanion.org/handbook/G_12.html
>
> The output of this particular anova() will include a value named
> 'LR.stat', the likelihood ratio statistic. The LR.stat is twice the
> difference of each log-likelihood (absolute value) of the respective
> models. Is LR.stat the mixed-model version of an "effect size"? If so, how
> does one determine if the effect is small, large, in-between, etc?
>
> Cheers,
> Sal
>
> Salvatore A. Sidoti
> PhD Candidate
> Behavioral Ecology
> The Ohio State University
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.