Setting Up a Project for Inter-coder Agreement Analysis

You can test inter-coder agreement in text, audio and video documents. Image documents are not supported. This also applies to image quotations in text PDF documents.

When you want to run an inter-coder agreement analysis, you are working in a team. Therefore, you need to know how to set up a team project, distribute and collect team projects for merging. Please read the chapter on Team Work for further information. Here, you find the information that is specific to inter-coder agreement analysis only.

Project Setup for ICA Analysis

As you need to start from a common Master project, you cannot perform an ICA analysis if one of the coders is using the Web version. All coders need to use the desktop version (Mac or Windows). For further information see Merging Projects.

Instructions for the Principal Investigator (PI)

  • The PI develops the code system adding clear definitions for each code. This means, the PI has likely coded the data as well. See also Requirements for Coding. This is the Master Project.

  • Based on the master project, the PI creates a sub-project for the ICA analysis. This is easiest be done by creating a project snapshot. Usually, the coders for the ICA analysis code 10-20% of the material. Another approach is the number of codings you need to achieve a certain level of significance. See Sample Size and Decision Rules.

  • The PI deletes all documents from the snapshot project that coders should not code. See Document Management.

Option A

Next, the PI can either remove all quotations, so that the snapshot project for the coders only contain the documents that need to be coded, and the codes with the definitions. The coders then need to find and mark text segments they find relevant and apply one or more codes from the list. This option is best used for longer documents where you are interested in seeing whether other coders find the same passages relevant for a given topic. For example, you have a number of reports, and the aim is to identify all segments related to business strategies, labour laws and gender issues. If this is the question of interest, it is necessary that all coders set quotations themselves (Option A: Delete all Quotations).

The PI deletes all quotations: Open the Quotation Manager, select all quotations and click on the Delete button.

The PI saves the project, creates a project bundle file and sends it to the coders.

Option B

Another option is to leave the quotations in the project, but to remove all codings. This means, the coders get a project that contains the documents to be coded, pre-defined quotations and a list of codes. This is useful if the aim is to see whether different coders understand the codes in the same way and apply them to the same data segments. This assures that the code system is replicable and that also others see the same or similar things in the data as the principal investigator (Option B: Remove all codings).

The PI removes only the codings: Open the Document Manager, select all documents, click on the Tools tab and from there select Remove codings.
The PI saves the project, creates a project bundle file and sends it to the coders.

Instructions f

An important requirement for inter-coder agreement analysis is the independence of the coders. Thus, the person who has developed the code system cannot be one of the coders whose coding goes into the ICA analysis.

It is recommended to provide the coders with the following instructions.

  • As coder, you receive a project bundle file from the principal investigator.

  • When importing the project, add your name or initials to the project name.

  • After opening the project, double-check your user name.

  • Open the Code Manager and familiarize yourself with the code system. Read all code definitions.

  • Depending on the aim of the analysis and how the project was set up, you either need to set your own quotations and apply the codes; or you need to apply codes to pre-defined quotations.

  • When applying codes, never apply more than one code from a specific semantic domain to a single or overlapping quotations. This is referred to as mutual exclusive coding.

  • If fitting you can apply codes from multiple semantic domains to the same or overlapping quotations (multi-valued coding).

  • Do not make any changes to the codes - do not alter the code definition, do not change the code name, or the code color.

  • If you do not understand a code definition, or find a code label not fitting, create a memo and name it 'Comments from coder (yourname)'. Write all of your comments, issues you find and ideas you have into this memo.

  • Do not consult with other coders. This is important for your coding to remain unbiased. The data must be coded by all coders independently.

  • Once you are done coding, create a project bundle file and send it back to the principal investigator.

Continues Workflow

  • Once the coders are done with they work, they prepare a project bundle file and send it back to the PI.

  • The PI imports the project bundle files from each coder, checks the work of each coder and reads the memo they have written (if any).

  • Next the projects need to be merged. See Merging projects for ICA analysis.

  • Then the PI can run the ICA analysis.