Running an ICA Analysis

A good time to have your coding checked by other coders is when you have built a stable code system and all codes are defined. This means, this is somewhere in the middle of the coding process. Once a satisfactory ICA coefficient is achieved, the principal investigator has the assurance that his or her codes can be understood and applied by others and can continue to work with the code system.

From the main menu, select Analysis > Calculate Intercoder Agreement.

Click on the Add User button and select two or more coders.

Click on the Add Documents button and select the documents that should be included in the analysis. Only those documents are listed that have been coded by all selected coders.

Click on the Add Semantic Domain button and add one or more codes to the domain. Alternatively, you can drag and drop codes from the Project Explorer or the code list into the semantic domain field. This works best, if you place the ICA window next to the main window.

Repeat this process for each semantic domain.

If you are not sure, what a semantic domain is, you find more information here.

Whether you add semantic domains or only a single code in place of a domain, this has an effect on the type of analysis you can run. If you only add one code per semantic domain, you cannot calculate Cu/cu-alpha coefficients. See Methods for Testing ICA.

After you have added coders, documents and codes, your screen looks similar to what is shown in:

Entering coders, codes and documents

  • The documents are represented by a line for each coder and code. If you have added multiple documents into the analysis, you see a black vertical line that marks the beginning of the next document.

  • The quotations for the code are marked on the line in the same color as the coder icon.

  • To the right of each coder line you see some descriptive statistics: The coder Susanne (yellow) has applied the code "activities clubbing / going for a drink" 11 times. The number of characters that are coded with this code are 1331 out of a total of 172.404 characters in all three selected documents, which is 0,8%. The coder Sabine (red) has applied the code "activities clubbing / going for a drink" 8 times. The number of characters that are coded with this code are 1057 out of a total of 172.404 characters in all three selected documents, which is 0,6%.

On the right-hand side you see the Quotation Reader, which displays all quotations of all selected document and codes.

  • If you click on the area with the two lines for the two coders for one code, only the quotations of the selected code are shown.

  • If you select a semantic domain, only the quotations of the domain are shown in the Quotation Reader.

You can remove coders, codes or documents from the analysis by clicking on the x.

Calculating an ICA Coefficient

To calculate ICA, click on the button for Agreement Measures in the toolbar (next to the View button) and select one of the four methods. See Methods For Testing ICA for more detail.

Select ICA coefficient

Selecting the Krippendorff's Cu /cu-alpha coefficient for the above example, yields the following results:

Results: Krippendorff's CU/cu alpha