Generating a model fitness when score using svyglm?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Generating a model fitness when score using svyglm?

Brad Fulton
Does anyone know how to calculated a BIC score (or an equivalent model fitness score) when using svyglm for logistic regressions?

Thanks

Brad
Reply | Threaded
Open this post in threaded view
|

Re: Generating a model fitness when score using svyglm?

Thomas Lumley


On Wed, 28 Apr 2010, Brad Fulton wrote:

> Does anyone know how to calculated a BIC score (or an equivalent model
> fitness score) when using svyglm for logistic regressions?
>

No.  That is, the model is not fitted by maximum likelihood, so BIC doesn't approximate posterior probabilities.

Now, the deviance returned by svyglm() is scaled to the sample size, so if the survey design isn't informative it should be more or less in the same ballpark as a deviance from an independent sample, and the usual BIC calculation might give somewhat helpful results.

    -thomas

Thomas Lumley Assoc. Professor, Biostatistics
[hidden email] University of Washington, Seattle

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Generating a model fitness score when using svyglm?

Brad Fulton
So are you saying that one way to estimate goodness of fit would be to run each of models using glm() and compare their BIC scores?

Is there a recommended way to demonstrate improvements in goodness of fit when using svyglm?

Thanks

Brad
Reply | Threaded
Open this post in threaded view
|

Re: Generating a model fitness score when using svyglm?

Thomas Lumley
On Wed, 28 Apr 2010, Brad Fulton wrote:

> Is there a recommended way to demonstrate improvements in goodness of fit
> when using svyglm?
>

No.

But then, I may be the wrong person to ask, since I wouldn't use AIC, BIC, CIC, DIC, EIC,.... for independently sampled data either.

In my view, you are either doing prediction, in which case I wouldn't trust the BIC penalty function (or any other fixed penalty) and would want out-of-sample prediction error, or doing inference about effects, in which case BIC is the wrong criterion entirely.


If you would use BIC on independently sampled data, then applying the same formula to svyglm() output will give a reasonable approximation to the same thing, but I wouldn't put much weight in small differences.

     -thomas


Thomas Lumley Assoc. Professor, Biostatistics
[hidden email] University of Washington, Seattle

______________________________________________
[hidden email] mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.