troubleshooting data structure to run krippendorff's alpha

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

troubleshooting data structure to run krippendorff's alpha

Hallie Kamesch
Hi -
I'm trying to run Krippendorff's alpha for data consisting of 4 subjects
rated on 6 events each by three raters.  The ratings are interval ratio
scale data.

I've rearranged my data into a 3 x 24  of ratersXevents. (per this
discussion on CrossValidated: (
https://stats.stackexchange.com/questions/255164/inter-rater-reliability-for-binomial-repeated-ratings-from-two-or-more-raters/256144#256144)
).

This is the code I've used:
library(irr)
dat <- read.csv(file.choose(), header = TRUE)
head(dat)
kripp.alpha(dat, method=c("ratio"))
#### error message: Error in sort.list(y) : 'x' must be atomic for
'sort.list'
Have you called 'sort' on a list?
kripp.alpha(dat,"ratio")
#### error message: Error in sort.list(y) : 'x' must be atomic for
'sort.list'
Have you called 'sort' on a list?

I read rhelp on sort, but I'm still confused.  Please help!
Thank you!

PS
I arranged my data in that matrix based upon this comment and response from
the CrossValidated posting forum (
https://stats.stackexchange.com/questions/255164/inter-rater-reliability-for-binomial-repeated-ratings-from-two-or-more-raters/256144#256144),
but my question above was rejected there.

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: troubleshooting data structure to run krippendorff's alpha

Jeff Newmiller
I don't understand most of what you wrote, but when you say "matrix" you are mistaken. A matrix is NOT the same thing as a data frame, which is what you get when you call read.csv(). Read

RShowDoc("R-intro")

Sections 5 and 6... A data frame is a list of column vectors, while a matrix is a vector with a dimension attribute.

You can use the as.matrix function to convert a data frame to a matrix.

On January 27, 2019 9:08:02 AM PST, Hallie Kamesch <[hidden email]> wrote:

>Hi -
>I'm trying to run Krippendorff's alpha for data consisting of 4
>subjects
>rated on 6 events each by three raters.  The ratings are interval ratio
>scale data.
>
>I've rearranged my data into a 3 x 24  of ratersXevents. (per this
>discussion on CrossValidated: (
>https://stats.stackexchange.com/questions/255164/inter-rater-reliability-for-binomial-repeated-ratings-from-two-or-more-raters/256144#256144)
>).
>
>This is the code I've used:
>library(irr)
>dat <- read.csv(file.choose(), header = TRUE)
>head(dat)
>kripp.alpha(dat, method=c("ratio"))
>#### error message: Error in sort.list(y) : 'x' must be atomic for
>'sort.list'
>Have you called 'sort' on a list?
>kripp.alpha(dat,"ratio")
>#### error message: Error in sort.list(y) : 'x' must be atomic for
>'sort.list'
>Have you called 'sort' on a list?
>
>I read rhelp on sort, but I'm still confused.  Please help!
>Thank you!
>
>PS
>I arranged my data in that matrix based upon this comment and response
>from
>the CrossValidated posting forum (
>https://stats.stackexchange.com/questions/255164/inter-rater-reliability-for-binomial-repeated-ratings-from-two-or-more-raters/256144#256144),
>but my question above was rejected there.
>
> [[alternative HTML version deleted]]
>
>______________________________________________
>[hidden email] mailing list -- To UNSUBSCRIBE and more, see
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide
>http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.

--
Sent from my phone. Please excuse my brevity.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: troubleshooting data structure to run krippendorff's alpha

Jim Lemon-4
In reply to this post by Hallie Kamesch
Hi Halllie,
As Jeff noted, a data frame is not a matrix (it is a variety of list),
so that looks like your problem.

hkdf<-data.frame(sample(3:5,4,TRUE),sample(1:3,4,TRUE),sample(2:4,4,TRUE),
 sample(3:5,4,TRUE),sample(1:3,4,TRUE),sample(2:4,4,TRUE))
library(irr)
kripp.alpha(hkdf)
kripp.alpha(as.matrix(hkdf))

Jim

On Mon, Jan 28, 2019 at 6:04 PM Hallie Kamesch <[hidden email]> wrote:

>
> Hi -
> I'm trying to run Krippendorff's alpha for data consisting of 4 subjects
> rated on 6 events each by three raters.  The ratings are interval ratio
> scale data.
>
> I've rearranged my data into a 3 x 24  of ratersXevents. (per this
> discussion on CrossValidated: (
> https://stats.stackexchange.com/questions/255164/inter-rater-reliability-for-binomial-repeated-ratings-from-two-or-more-raters/256144#256144)
> ).
>
> This is the code I've used:
> library(irr)
> dat <- read.csv(file.choose(), header = TRUE)
> head(dat)
> kripp.alpha(dat, method=c("ratio"))
> #### error message: Error in sort.list(y) : 'x' must be atomic for
> 'sort.list'
> Have you called 'sort' on a list?
> kripp.alpha(dat,"ratio")
> #### error message: Error in sort.list(y) : 'x' must be atomic for
> 'sort.list'
> Have you called 'sort' on a list?
>
> I read rhelp on sort, but I'm still confused.  Please help!
> Thank you!
>
> PS
> I arranged my data in that matrix based upon this comment and response from
> the CrossValidated posting forum (
> https://stats.stackexchange.com/questions/255164/inter-rater-reliability-for-binomial-repeated-ratings-from-two-or-more-raters/256144#256144),
> but my question above was rejected there.
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: troubleshooting data structure to run krippendorff's alpha

Hallie Kamesch
Hi all,
Thank you for your responses. You are correct that it is not a matrix. I used the incorrect term.
I meant I put my data in a spreadsheet with three rows and 24 columns.

Sent from my iPhone

> On Jan 28, 2019, at 3:36 AM, Jim Lemon <[hidden email]> wrote:
>
> Hi Halllie,
> As Jeff noted, a data frame is not a matrix (it is a variety of list),
> so that looks like your problem.
>
> hkdf<-data.frame(sample(3:5,4,TRUE),sample(1:3,4,TRUE),sample(2:4,4,TRUE),
> sample(3:5,4,TRUE),sample(1:3,4,TRUE),sample(2:4,4,TRUE))
> library(irr)
> kripp.alpha(hkdf)
> kripp.alpha(as.matrix(hkdf))
>
> Jim
>
>> On Mon, Jan 28, 2019 at 6:04 PM Hallie Kamesch <[hidden email]> wrote:
>>
>> Hi -
>> I'm trying to run Krippendorff's alpha for data consisting of 4 subjects
>> rated on 6 events each by three raters.  The ratings are interval ratio
>> scale data.
>>
>> I've rearranged my data into a 3 x 24  of ratersXevents. (per this
>> discussion on CrossValidated: (
>> https://stats.stackexchange.com/questions/255164/inter-rater-reliability-for-binomial-repeated-ratings-from-two-or-more-raters/256144#256144)
>> ).
>>
>> This is the code I've used:
>> library(irr)
>> dat <- read.csv(file.choose(), header = TRUE)
>> head(dat)
>> kripp.alpha(dat, method=c("ratio"))
>> #### error message: Error in sort.list(y) : 'x' must be atomic for
>> 'sort.list'
>> Have you called 'sort' on a list?
>> kripp.alpha(dat,"ratio")
>> #### error message: Error in sort.list(y) : 'x' must be atomic for
>> 'sort.list'
>> Have you called 'sort' on a list?
>>
>> I read rhelp on sort, but I'm still confused.  Please help!
>> Thank you!
>>
>> PS
>> I arranged my data in that matrix based upon this comment and response from
>> the CrossValidated posting forum (
>> https://stats.stackexchange.com/questions/255164/inter-rater-reliability-for-binomial-repeated-ratings-from-two-or-more-raters/256144#256144),
>> but my question above was rejected there.
>>
>>        [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: troubleshooting data structure to run krippendorff's alpha

Hallie Kamesch
Thank you Jim, for the code, and thank you Jeff for the tutorial PDF.  I've
read through the sections and I appreciate the help.
I'm in way over my head - I don't even understand enough of the vocabulary
to ask my question correctly.
Jim, in your code, I ended up with an entry of 4 observations of 6
variables. I understand how that happened now since I read your code - that
helped very much.
My only problem, that I can't figure out, is how to make it so I have 3
raters with 4 observations of 6 variables.
I really am trying to educate myself enough to not waste your time:  I've
?data.frame, ?sample, ?matrix, ?$names, ?attributes, etc... I read the
sections in Jeff's PDF, and the tutorials on datamentor, I'm just not
finding how to do this. I'm sorry this is such a newbie question.
thank you for your time,
hallie

On Mon, Jan 28, 2019 at 1:21 PM Hallie Kamesch <[hidden email]>
wrote:

> Hi all,
> Thank you for your responses. You are correct that it is not a matrix. I
> used the incorrect term.
> I meant I put my data in a spreadsheet with three rows and 24 columns.
>
> Sent from my iPhone
>
> > On Jan 28, 2019, at 3:36 AM, Jim Lemon <[hidden email]> wrote:
> >
> > Hi Halllie,
> > As Jeff noted, a data frame is not a matrix (it is a variety of list),
> > so that looks like your problem.
> >
> >
> hkdf<-data.frame(sample(3:5,4,TRUE),sample(1:3,4,TRUE),sample(2:4,4,TRUE),
> > sample(3:5,4,TRUE),sample(1:3,4,TRUE),sample(2:4,4,TRUE))
> > library(irr)
> > kripp.alpha(hkdf)
> > kripp.alpha(as.matrix(hkdf))
> >
> > Jim
> >
> >> On Mon, Jan 28, 2019 at 6:04 PM Hallie Kamesch <
> [hidden email]> wrote:
> >>
> >> Hi -
> >> I'm trying to run Krippendorff's alpha for data consisting of 4 subjects
> >> rated on 6 events each by three raters.  The ratings are interval ratio
> >> scale data.
> >>
> >> I've rearranged my data into a 3 x 24  of ratersXevents. (per this
> >> discussion on CrossValidated: (
> >>
> https://stats.stackexchange.com/questions/255164/inter-rater-reliability-for-binomial-repeated-ratings-from-two-or-more-raters/256144#256144
> )
> >> ).
> >>
> >> This is the code I've used:
> >> library(irr)
> >> dat <- read.csv(file.choose(), header = TRUE)
> >> head(dat)
> >> kripp.alpha(dat, method=c("ratio"))
> >> #### error message: Error in sort.list(y) : 'x' must be atomic for
> >> 'sort.list'
> >> Have you called 'sort' on a list?
> >> kripp.alpha(dat,"ratio")
> >> #### error message: Error in sort.list(y) : 'x' must be atomic for
> >> 'sort.list'
> >> Have you called 'sort' on a list?
> >>
> >> I read rhelp on sort, but I'm still confused.  Please help!
> >> Thank you!
> >>
> >> PS
> >> I arranged my data in that matrix based upon this comment and response
> from
> >> the CrossValidated posting forum (
> >>
> https://stats.stackexchange.com/questions/255164/inter-rater-reliability-for-binomial-repeated-ratings-from-two-or-more-raters/256144#256144
> ),
> >> but my question above was rejected there.
> >>
> >>        [[alternative HTML version deleted]]
> >>
> >> ______________________________________________
> >> [hidden email] mailing list -- To UNSUBSCRIBE and more, see
> >> https://stat.ethz.ch/mailman/listinfo/r-help
> >> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> >> and provide commented, minimal, self-contained, reproducible code.
>

        [[alternative HTML version deleted]]

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: troubleshooting data structure to run krippendorff's alpha

Jim Lemon-4
Hi Hallie,
If I understand your email correctly, you have four repeated
observations by the three raters of the same six variables. This is a
tougher problem and I can't solve it at the moment. I'll return to
this later and see if I can offer a solution.


Jim

On Wed, Jan 30, 2019 at 3:56 AM Hallie Kamesch <[hidden email]> wrote:
>
> Thank you Jim, for the code, and thank you Jeff for the tutorial PDF.  I've read through the sections and I appreciate the help.
> I'm in way over my head - I don't even understand enough of the vocabulary to ask my question correctly.
> Jim, in your code, I ended up with an entry of 4 observations of 6 variables. I understand how that happened now since I read your code - that helped very much.
> My only problem, that I can't figure out, is how to make it so I have 3 raters with 4 observations of 6 variables.
> I really am trying to educate myself enough to not waste your time:  I've ?data.frame, ?sample, ?matrix, ?$names, ?attributes, etc... I read the sections in Jeff's PDF, and the tutorials on datamentor, I'm just not finding how to do this. I'm sorry this is such a newbie question.
> thank you for your time,
> hallie
>

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Reply | Threaded
Open this post in threaded view
|

Re: troubleshooting data structure to run krippendorff's alpha

Jim Lemon-4
Hi Hallie,
I tried both the "cccUst" and "cccvc" functions in the "cccrm"
package. While I can get what looks like sensible statistics with the
following example, I am not sure that it can be interpreted as you
wish. For one thing, it assumes that the concordance will be the same
on all variables. I was not able to get a statistic on each variable
separately. Perhaps someone who is more familiar with the package can
offer better advice. Also, check the "hkdf" data frame to ensure that
it looks like your data.

hkdf<-data.frame(rater=rep(1:3,each=24),occasion=rep(rep(1:4,each=6),3),
 var=rep(rep(1:6,4),3),obs=runif(72))
library(cccrm)
cccUst(hkdf,"obs","rater","occasion")
cccvc(hkdf,"obs","rater","occasion")

Jim

On Wed, Jan 30, 2019 at 1:01 PM Jim Lemon <[hidden email]> wrote:
>
> Hi Hallie,
> If I understand your email correctly, you have four repeated
> observations by the three raters of the same six variables. This is a
> tougher problem and I can't solve it at the moment. I'll return to
> this later and see if I can offer a solution.
>
>
> Jim

______________________________________________
[hidden email] mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.