Jump to content

Inter-rater reliability between two projects in NVivo


Guest axg928@case.edu

Recommended Posts

Guest axg928@case.edu

I am trying to measure inter-rater reliability between two different coders who have coded the same set of interviews separately, using their respective NVivo accounts, (so essentially, there are two separate projects, one from coder A, and one from coder B, both containing the same files same codes, but they have been coded separately).

I tried going through the steps listed in this document to conduct a coding comparison query am a little confused as to how this would be able to measure agreement between the two projects. As listed towards the bottom of the document, when running the query all the Kappa  coefficients are either 0 or 1, so I know something wrong. 

I have also tried merging the two projects together into one new project, and then running the coding query again. This resulted in coefficients of all 0s and 1s, which is also not accurate. I noticed that there are some discrepancies in the way the files and nodes are organized (even though the codes are the same) between Coder As project and Coder Bs project.  

For a third attempt, I tried to consolidate the Node and file classifications in both projects (A & B), and then merged them again, and then ran the Coding comparison query. In doing this, I got a Kappa coefficient of 1s across everything, which also can not be correct. Is there something that I am missing/a directory/website/resource that would help me walk through this process of conducting inter-rater reliability between these two projects? Please help and thank you in advance!! 

Link to post
Share on other sites

Hello axg928,

If coders worked on separate projects, the correct way to calculate ICR is merging two projects. 

Having said that, I would suggest to have a look below information to be sure that two projects merged appropriately.

All my Kappa coefficients are 0 or 1. Is something wrong?

This most often indicates that one of the two users being compared has not coded any of the selected files at the selected nodes.

In your Coding Comparison query results:

  • If the columns “A and B (%)” and “A and Not B (%)” are both entirely full of zeros, then user A has not coded any of the files at the selected nodes
  • If the columns “A and B (%)” and “B and Not A (%)” are both entirely full of zeros, then user B has not coded any of the files at the selected nodes

If the user’s work has been imported from another NVivo project, this may indicate that this user’s coding has not been imported correctly and needs to be re-imported.

When using the Import Project feature in NVivo to import coding:

  • Be sure to choose the option “Merge into existing item” for duplicate items
  • When selecting which data to import, do not choose the option “Selected (excluding content)”
  • If you choose the option “Selected (including content)”, be sure to select to import “Coding” in the Import Options dialog.

Hope this helps,

Link to post
Share on other sites
Guest axg928

Hi Canarik, 

Thank you for your response. I have tried merging the two projects as you suggested above and re-run the coding but I have a feeling that I am still not merging correctly because other than just a few rows which do now have a Kappa Coefficient that is neither 0 nor 1 (although the coefficients are actually surprisingly low), and when looking at the "A and B (%)" columns along with the "A and Not B(%)" and "B and Not A(%)" columns, it indicates to me that the users A and B have not coded any of the selected nodes, which is not the case. 

When I look at the newly merged documents, it looks like even though I am making sure to check the "Merge into existing item" for duplicate items, it is creating a duplicate for most of the folder. My process has been to make a copy of Coder A's project (so I have original on hand), then use the Import tab to import Coder B's project into Coder A's project.  I have tried to merge in the following ways: 

  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: All (including content) --> For duplicate items: Merge into existing item
  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: Selected (including content)--> Files, Nodes, Cases--> For duplicate items: Merge into existing item

For both versions of the merged documents, I then tried completing the coding comparison query through Explore tab --> Coding Comparison but as I mentioned above, I am not getting accurate coefficients.  Is there a different way to import and merge and do the coding query or am I missing something? 

Thank you again for your help, I really appreciate it. 

Link to post
Share on other sites
Guest axg928

Hi Canarik, 

Thank you for your response. I have tried merging the two projects as you suggested above and re-run the coding but I have a feeling that I am still not merging correctly because other than just a few rows which do now have a Kappa Coefficient that is neither 0 nor 1 (although the coefficients are actually surprisingly low), and when looking at the "A and B (%)" columns along with the "A and Not B(%)" and "B and Not A(%)" columns, it indicates to me that the users A and B have not coded any of the selected nodes, which is not the case. 

When I look at the newly merged documents, it looks like even though I am making sure to check the "Merge into existing item" for duplicate items, it is creating a duplicate for most of the folder. My process has been to make a copy of Coder A's project (so I have original on hand), then use the Import tab to import Coder B's project into Coder A's project.  I have tried to merge in the following ways: 

  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: All (including content) --> For duplicate items: Merge into existing item
  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: Selected (including content)--> Files, Nodes, Cases--> For duplicate items: Merge into existing item

For both versions of the merged documents, I then tried completing the coding comparison query through Explore tab --> Coding Comparison but as I mentioned above, I am not getting accurate coefficients.  Is there a different way to import and merge and do the coding query or am I missing something? 

Thank you again for your help, I really appreciate it. 

Link to post
Share on other sites
Guest axg928

Hi Canarik, 

Thank you for your response. I have tried merging the two projects as you suggested above and re-run the coding but I have a feeling that I am still not merging correctly because other than just a few rows which do now have a Kappa Coefficient that is neither 0 nor 1 (although the coefficients are actually surprisingly low), and when looking at the "A and B (%)" columns along with the "A and Not B(%)" and "B and Not A(%)" columns, it indicates to me that the users A and B have not coded any of the selected nodes, which is not the case. 

When I look at the newly merged documents, it looks like even though I am making sure to check the "Merge into existing item" for duplicate items, it is creating a duplicate for most of the folder. My process has been to make a copy of Coder A's project (so I have original on hand), then use the Import tab to import Coder B's project into Coder A's project.  I have tried to merge in the following ways: 

  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: All (including content) --> For duplicate items: Merge into existing item
  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: Selected (including content)--> Files, Nodes, Cases--> For duplicate items: Merge into existing item

For both versions of the merged documents, I then tried completing the coding comparison query through Explore tab --> Coding Comparison but as I mentioned above, I am not getting accurate coefficients.  Is there a different way to import and merge and do the coding query or am I missing something? 

Thank you again for your help, I really appreciate it. 

Link to post
Share on other sites
Guest axg928

Hi Canarik, 

Thank you for your response. I have tried merging the two projects as you suggested above and re-run the coding but I have a feeling that I am still not merging correctly because other than just a few rows which do now have a Kappa Coefficient that is neither 0 nor 1 (although the coefficients are actually surprisingly low), and when looking at the "A and B (%)" columns along with the "A and Not B(%)" and "B and Not A(%)" columns, it indicates to me that the users A and B have not coded any of the selected nodes, which is not the case. 

When I look at the newly merged documents, it looks like even though I am making sure to check the "Merge into existing item" for duplicate items, it is creating a duplicate for most of the folder. My process has been to make a copy of Coder A's project (so I have original on hand), then use the Import tab to import Coder B's project into Coder A's project.  I have tried to merge in the following ways: 

  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: All (including content) --> For duplicate items: Merge into existing item
  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: Selected (including content)--> Files, Nodes, Cases--> For duplicate items: Merge into existing item

For both versions of the merged documents, I then tried completing the coding comparison query through Explore tab --> Coding Comparison but as I mentioned above, I am not getting accurate coefficients.  Is there a different way to import and merge and do the coding query or am I missing something? 

Thank you again for your help, I really appreciate it. 

Link to post
Share on other sites
Guest axg928

Hi Canarik, 

Thank you for your response. I have tried merging the two projects as you suggested above and re-run the coding but I have a feeling that I am still not merging correctly because other than just a few rows which do now have a Kappa Coefficient that is neither 0 nor 1 (although the coefficients are actually surprisingly low), and when looking at the "A and B (%)" columns along with the "A and Not B(%)" and "B and Not A(%)" columns, it indicates to me that the users A and B have not coded any of the selected nodes, which is not the case. 

When I look at the newly merged documents, it looks like even though I am making sure to check the "Merge into existing item" for duplicate items, it is creating a duplicate for most of the folder. My process has been to make a copy of Coder A's project (so I have original on hand), then use the Import tab to import Coder B's project into Coder A's project.  I have tried to merge in the following ways: 

  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: All (including content) --> For duplicate items: Merge into existing item
  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: Selected (including content)--> Files, Nodes, Cases--> For duplicate items: Merge into existing item

For both versions of the merged documents, I then tried completing the coding comparison query through Explore tab --> Coding Comparison but as I mentioned above, I am not getting accurate coefficients.  Is there a different way to import and merge and do the coding query or am I missing something? 

Thank you again for your help, I really appreciate it. 

Link to post
Share on other sites
Guest axg928

Hi Canarik, 

Thank you for your response. I have tried merging the two projects as you suggested above and re-run the coding but I have a feeling that I am still not merging correctly because other than just a few rows which do now have a Kappa Coefficient that is neither 0 nor 1 (although the coefficients are actually surprisingly low), and when looking at the "A and B (%)" columns along with the "A and Not B(%)" and "B and Not A(%)" columns, it indicates to me that the users A and B have not coded any of the selected nodes, which is not the case. 

When I look at the newly merged documents, it looks like even though I am making sure to check the "Merge into existing item" for duplicate items, it is creating a duplicate for most of the folder. My process has been to make a copy of Coder A's project (so I have original on hand), then use the Import tab to import Coder B's project into Coder A's project.  I have tried to merge in the following ways: 

  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: All (including content) --> For duplicate items: Merge into existing item
  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: Selected (including content)--> Files, Nodes, Cases--> For duplicate items: Merge into existing item

For both versions of the merged documents, I then tried completing the coding comparison query through Explore tab --> Coding Comparison but as I mentioned above, I am not getting accurate coefficients.  Is there a different way to import and merge and do the coding query or am I missing something? 

Thank you again for your help, I really appreciate it. 

Link to post
Share on other sites
Guest axg928

Hi Canarik, 

Thank you for your response. I have tried merging the two projects as you suggested above and re-run the coding but I have a feeling that I am still not merging correctly because other than just a few rows which do now have a Kappa Coefficient that is neither 0 nor 1 (although the coefficients are actually surprisingly low), and when looking at the "A and B (%)" columns along with the "A and Not B(%)" and "B and Not A(%)" columns, it indicates to me that the users A and B have not coded any of the selected nodes, which is not the case. 

When I look at the newly merged documents, it looks like even though I am making sure to check the "Merge into existing item" for duplicate items, it is creating a duplicate for most of the folder. My process has been to make a copy of Coder A's project (so I have original on hand), then use the Import tab to import Coder B's project into Coder A's project.  I have tried to merge in the following ways: 

  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: All (including content) --> For duplicate items: Merge into existing item
  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: Selected (including content)--> Files, Nodes, Cases--> For duplicate items: Merge into existing item

For both versions of the merged documents, I then tried completing the coding comparison query through Explore tab --> Coding Comparison but as I mentioned above, I am not getting accurate coefficients.  Is there a different way to import and merge and do the coding query or am I missing something? 

Thank you again for your help, I really appreciate it. 

Link to post
Share on other sites
Guest axg928

Hi Canarik, 

Thank you for your response. I have tried merging the two projects as you suggested above and re-run the coding but I have a feeling that I am still not merging correctly because other than just a few rows which do now have a Kappa Coefficient that is neither 0 nor 1 (although the coefficients are actually surprisingly low), and when looking at the "A and B (%)" columns along with the "A and Not B(%)" and "B and Not A(%)" columns, it indicates to me that the users A and B have not coded any of the selected nodes, which is not the case. 

When I look at the newly merged documents, it looks like even though I am making sure to check the "Merge into existing item" for duplicate items, it is creating a duplicate for most of the folder. My process has been to make a copy of Coder A's project (so I have original on hand), then use the Import tab to import Coder B's project into Coder A's project.  I have tried to merge in the following ways: 

  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: All (including content) --> For duplicate items: Merge into existing item
  • Coder A_Project Copy, Import --> Project (Coder B Project)--> Select the project items to import: Selected (including content)--> Files, Nodes, Cases--> For duplicate items: Merge into existing item

For both versions of the merged documents, I then tried completing the coding comparison query through Explore tab --> Coding Comparison but as I mentioned above, I am not getting accurate coefficients.  Is there a different way to import and merge and do the coding query or am I missing something? 

Thank you again for your help, I really appreciate it. 

Link to post
Share on other sites
Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...