Help with Krippendorff's Alpha

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Help with Krippendorff's Alpha

JRViolet
Hello!

I am requesting help for a data set in which I have 6 evaluators (I've called them 'Trainers'), each of which took a questionnaire with 180 questions. The possible answers to the questions are ordinal numbers. I am looking to test inter-rater reliability of the questionnaire using Krippendorff's alpha. I have already run this in R, however when I get the output it seems the matrix is inverted. While it seems like it should be an easy problem to fix I don't see how I would be able to create my matrix the other way without typing out all 180 scores manually. For reference, I'm using the irr package to use kripp.alpha().

> Trainers<-matrix(c(Trainer_1,Trainer_2, Trainer_3, Trainer_5, Trainer_6, Trainer_12), ncol=6)
> kripp.alpha(Trainers, method=c("ordinal"))
Krippendorff's alpha
Subjects = 6
Raters = 180
alpha = -0.00149

While I'm getting an output - shouldn't my raters be 6 and my subjects be 180?

Appreciate the help!

JR

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Help with Krippendorff's Alpha

Bert Gunter-2
Well,....

"Appreciate the help!"

Indeed, you should for any that you receive. But do note per the posting
guide linked below:

"For questions about functions in standard packages distributed with R (see
the FAQ Add-on packages in R
<http://cran.r-project.org/doc/FAQ/R-FAQ.html#Add-on-packages-in-R>), ask
questions on R-help.
If the question relates to a *contributed package* , e.g., one downloaded
from CRAN, try contacting the package maintainer first. You can also use
find("functionname") and packageDescription("packagename") to find this
information. *Only* send such questions to R-help or R-devel if you get no
reply or need further assistance. This applies to both requests for help
and to bug reports."

So while optimism is often a virtue, don't be disappointed if it is not
rewarded here. For context, there are in excess of 20,000 R packages out
there, so expectations of help on any of them is a long shot.

Bert Gunter

"The trouble with having an open mind is that people keep coming along and
sticking things into it."
-- Opus (aka Berkeley Breathed in his "Bloom County" comic strip )


On Fri, Jan 22, 2021 at 3:24 PM Jenn Russell <[hidden email]> wrote:

> Hello!
>
> I am requesting help for a data set in which I have 6 evaluators (I've
> called them 'Trainers'), each of which took a questionnaire with 180
> questions. The possible answers to the questions are ordinal numbers. I am
> looking to test inter-rater reliability of the questionnaire using
> Krippendorff's alpha. I have already run this in R, however when I get the
> output it seems the matrix is inverted. While it seems like it should be an
> easy problem to fix I don't see how I would be able to create my matrix the
> other way without typing out all 180 scores manually. For reference, I'm
> using the irr package to use kripp.alpha().
>
> > Trainers<-matrix(c(Trainer_1,Trainer_2, Trainer_3, Trainer_5, Trainer_6,
> Trainer_12), ncol=6)
> > kripp.alpha(Trainers, method=c("ordinal"))
> Krippendorff's alpha
> Subjects = 6
> Raters = 180
> alpha = -0.00149
>
> While I'm getting an output - shouldn't my raters be 6 and my subjects be
> 180?
>
> Appreciate the help!
>
> JR
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: Help with Krippendorff's Alpha

Jim Lemon-4
In reply to this post by JRViolet
Hi Jean,
kripp.alpha expects a classifier by object (in your case, score)
matrix as the first argument. If I read your code correctly you are
getting the scores zigzagging across six columns when you want a 6 x
180 matrix. My guess is that you want:

Trainers<-matrix(c(Trainer_1,Trainer_2, Trainer_3, Trainer_5,
Trainer_6, Trainer_12),
 nrow=6,byrow=TRUE)

Jim


On Sat, Jan 23, 2021 at 10:24 AM Jenn Russell <[hidden email]> wrote:

>
> Hello!
>
> I am requesting help for a data set in which I have 6 evaluators (I've called them 'Trainers'), each of which took a questionnaire with 180 questions. The possible answers to the questions are ordinal numbers. I am looking to test inter-rater reliability of the questionnaire using Krippendorff's alpha. I have already run this in R, however when I get the output it seems the matrix is inverted. While it seems like it should be an easy problem to fix I don't see how I would be able to create my matrix the other way without typing out all 180 scores manually. For reference, I'm using the irr package to use kripp.alpha().
>
> > Trainers<-matrix(c(Trainer_1,Trainer_2, Trainer_3, Trainer_5, Trainer_6, Trainer_12), ncol=6)
> > kripp.alpha(Trainers, method=c("ordinal"))
> Krippendorff's alpha
> Subjects = 6
> Raters = 180
> alpha = -0.00149
>
> While I'm getting an output - shouldn't my raters be 6 and my subjects be 180?
>
> Appreciate the help!
>
> JR
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.