Jump to content

Not getting Kappa values with code comparisons


Recommended Posts

Hello everyone, 

I am having issues calculating Kappa values for a project. There are two raters. The principle coder imported coder two's project into the master file and conducted a coding comparison query. After many tries, the kappa keeps coming back as either a zero or one (please see screenshot below)

We tried to troubleshoot with no luck. We were wondering if it's an issue with different operating systems (one mac and one window user). We followed the correct instructions to import the project according to this link: https://help-nv.qsrinternational.com/12/win/v12.1.101-d3ea61/Content/queries/coding-comparison-query.htm

Any input would be greatly appreciated.

Nada 

 

image.thumb.png.341834f950836ade2636e88b6bb39011.png

Link to post
Share on other sites

Hi Nada,

From the screenshot the kappa 0s are correct because if you look at the Disagreement percentages only coder B did any coding for the indicated codes. The kappa 1 is correct because neither of the coders coded for that code in the Twitter file. Have you double clicked any of the rows in the Coding Comparison chart to see the actual coding?

If this is not correct then it could be that the Twitter data was not merged. If you look in the Merged project and look at the Files area, is there only 1 Twitter data set or are there 2? If there are two, it has not merged because the Twitter files are slightly different.

The other thing is to  look at your codes. Are the codes in the two projects organized exactly the same in a hierarchy (if you used parent and child nodes). If they are not, even if the codes are exactly the same but are organized differently they will not  merge. If the problem is with the codes, you can cut and merge the duplicated codes in the Master project and then run the Coding Comparison Query again.

Best wishes,

 

Silvana

Link to post
Share on other sites
  • 2 weeks later...

Hello, 

I have had the same issue as Nada. And when I look at the coding stripes, it appears that one of the coders has a case ID attached to every single one of the coding references. In other words, it not only shows "(Codes\codename)" but also "(Cases\name of file\caseID)". Could this be the reason for obtaining kappas of zero for all the codes?

If so, is there a way to un-code all the case IDs from the text that has been coded?

Any help with this issue would be greatly appreciated!

Link to post
Share on other sites
On 5/20/2021 at 5:21 AM, Silvana said:

Hi Silvana,

Thank you so much for your reply. I am one of the researchers working on the project Nada mentions. In terms of your tips below, there is only one file and the node structure is the same. I actually think the root of the problem might be in the way each project was coded. So for my coding, it coded to cases/file/ID and the other researcher coded at node rather than cases. I am not sure why this might have happened other than my PC vs her mac treated the coding differently. Is there any way to fix this besides redoing the coding? 

Thanks, Melissa

 

Quote

 

 

Hi Nada,

From the screenshot the kappa 0s are correct because if you look at the Disagreement percentages only coder B did any coding for the indicated codes. The kappa 1 is correct because neither of the coders coded for that code in the Twitter file. Have you double clicked any of the rows in the Coding Comparison chart to see the actual coding?

If this is not correct then it could be that the Twitter data was not merged. If you look in the Merged project and look at the Files area, is there only 1 Twitter data set or are there 2? If there are two, it has not merged because the Twitter files are slightly different.

The other thing is to  look at your codes. Are the codes in the two projects organized exactly the same in a hierarchy (if you used parent and child nodes). If they are not, even if the codes are exactly the same but are organized differently they will not  merge. If the problem is with the codes, you can cut and merge the duplicated codes in the Master project and then run the Coding Comparison Query again.

Best wishes,

 

Silvana

 

Link to post
Share on other sites

Hi @Melissa MacKay,

I have the exact issue as you. My coding seems to have been coded to Cases as well as to Nodes, while my colleague's coding just seems to be at the Nodes. We were both working on the same Windows-based computer and started off with copies of the same master project, so not sure why my coding was coded to the cases\filename\case#.

Did you find a way to un-code all the case IDs from the text that has been coded? Or what the implications of un-coding from cases would be?

Thanks.

CP

Link to post
Share on other sites

Hi @Melissa MacKay and @CSP,

You don't want to uncode from the case - because that will limit what you can do in the queries. I am assuming here by case you mean the respondent case. However, did one of you create your thematic codes (Nodes) as cases? If so, just cut them and merge into the appropriate code (Node). Then you can run the kappa. It is hard to figure out what you did without seeing your projects.

Best wishes,

Silvana

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...

Important Information

Privacy Policyhttps://www.qsrinternational.com/privacy-policyhis site, you agree to our Terms of Use.