How Agreement and Disagreement is Measured

The coefficients that are available measure the extent of agreement or disagreement of different coders. Based on this measure one can infer reliability, but the coefficients do not measure reliability directly. Therefore: what is measured is inter-coder agreement and not inter-coder reliability.

Agreement is what we measure; reliability is what we infer from it.

Suppose two coders are given a text and code it following the provided instructions. After they are done, they give you the following record of their coding:

Coding example - Textual Continuum

  • There is agreement on the first pair of segments in terms of length and location (agreement: 10 units).

  • On the second pair they agree in length but not in location. There is agreement on an intersection of the data (agreement; 12 units, disagreement: 2 + 2 units).

  • In the third pair, coder 'A' finds a segment relevant, which is not recognized by coder 'B' (disagreement: 4 units).

  • The largest disagreement is observed in the last pair of segments. Coder 'A' takes a narrower view than coder 'B' (agreement: 4 units, disagreement: 6 units).

In terms of your ATLAS.ti coding, you need to think of your document as a textual continuum. Each character of your text is a unit of analysis for ICA, starting at character 1 and ending for instance at character 17500. For audio and video documents, the unit of analysis is a second. Images can currently not be used in an ICA analysis.

The quotation itself does not go into the analysis, but the characters or seconds that have been coded. In other words, if each coder creates quotations, it is not a total disagreement if they have not highlighted the exact same segment. The part of the quotations that overlap go into the analysis as agreement; the other parts as disagreement.

Another option is to work with pre-defined quotations. So that coders only need to apply the codes. For more information see Project Setup.

You can test inter-coder agreement in text, audio and video documents. Image documents are not supported. This also applies to image quotations in text PDF documents.

Example Application

Why are many business instilling a DevOps culture into their organization? written by: Jessica Diaz, Daniel López-Fernández, Jorge Perez and Ángel González-Prieto.