If you are not yet familiar with the coding process in ATLAS.ti, please see Coding Data
Sources for unreliable data are intra-coder inconsistencies and inter-coder disagreements.
To detect these, the coding process needs to be replicated. Replicability can be assured when several independently working coders (at least two) agree:
- on the use of the written coding instruction
- by highlighting the same textual segments to which the coding instructions apply
- by coding them using the same codes, or by identifying the same semantic domains that describe them and code them using the same codes for each semantic domain.
A semantic domain is defined as a set of distinct concepts that share common meanings. You can also think about them as a category with sub codes.
Examples of semantic domains are:
- emotions: joy
- emotions: excitement
- emotions: surprise
- emotions: sadness
- emotions: anger
- emotions: fear
- strategies: realize they are natural
- strategies: have a Plan B
- strategies: adjust expectations
- strategies: laugh it off
- strategies: get help
- actor: partner
- actor: mother
- actor: father
- actor: child
- actor: sibling
- actor: neighbour
- actor: colleague
Each semantic domain embraces mutually exclusive concepts indicated by a code. If you code a data segment with 'emotions: surprise', you cannot code it also 'emotions: fear'. If both are mentioned close to each other, you need to create two quotations and code them separately. You can however apply codes from different semantic domains to one quotation. See multi-valued coding below.
At times, it might be obvious from the domain name what the context is. The above sub code 'supporting each other' belongs to the context 'benefits of friendship', and it is not about work colleagues supporting each other. If the context is still unclear and could be interpreted in different ways, you need to make it unambiguous in the code definition.
Conceptual independence means:
- a sub code from one domain is only specific for this domain. It does not occur in any other domain.
- thus, each code only occurs once in the code system
This is also a rule that you should generally apply when building a code system. See Developing a Code System.
Therefore, as semantic domains are logically or conceptually independent of each other, it is possible to apply codes from different semantic domains to the same or overlapping quotations. For instance, In a section of your data, you may find something on the benefits of friendship, and some emotional aspects that are mentioned. As these codes come from different semantic domains (BENEFITS and EMOTIONS), they can both be applied.
Semantic domains can be developed deductively or inductively. In most studies applying a qualitative data analysis approach, development is likely to be inductive. This means, you develop the codes while you read the data step by step. For example, in an interview study about friendship, you may have coded some data segments 'caring for each other', 'supporting each other' and 'improve health and longevity'. Then you realize that these codes can be summarized on a higher level as 'BENEFITS OF FRIENDSHIP'. Thus, you set up a semantic domain BENEFITS with the sub codes:
- benefits: caring for each other
- benefits: supporting each other
- benefits: improve health and longevity
You continue coding and come across other segments that you code 'learning from each other'. As this also fits the domain BENEFITS, you add it to the domain by naming it:
- benefits: learning from each other
And so on. This way you can build semantic domains step by step inductively while developing the code system.
Once the code system is ready, and the code definitions are written in the code comment fields, you can prepare the project for inter-coder agreement testing. At this stage, you can no longer expand a semantic domain. See Project Setup.
When developing a code system, the aim is to cover the variability in the data so that no aspect that is relevant for the research question is left-out. This is referred to as exhaustiveness. On the domain level this means that all main topics are covered. On the sub code level, this means that the codes of a semantic domain cover all aspects of the domain, and the data can be sorted in the available codes without forcing them. An easy way out is to include a catch all 'miscellaneous' code for each domain into which coders can add all data that they think does not fit anywhere else. However, keep in mind that such catch-all codes will contribute little to answering the research questions.
- codes from one domain need to be applied in a mutually exclusive manner.
- codes from multiple semantic domains can be applied to the same or overlapping data segments.
Mutual exclusiveness: You can only apply one of the sub codes of a semantic domain to a quotation or to overlapping quotations. Using the same code colour for all codes of a semantic domain will help you to detect possible errors.
If you find that you have coded a quotation with two codes from the same semantic domain, you can fix it by splitting the quotation. This means, you change the length of the original quotation and create one new quotation, so you can apply the two codes to two distinct quotations.
If codes within a semantic domain are not applied in a mutually exclusive manner, the ICA coefficient is inflated -- and in the current implementation of the tool cannot be calculated!
Multi-Valued Coding: This means that you can apply multiple codes of different semantic domains to the same quotation. For instance, a respondent talks about anger in dealing with her mother and mentions that she had to adjust expectations, this can be coded with codes from the three semantic domains EMOTIONS; ACTOR and STRATEGIES.
Coding with codes from multiple semantic domains will allow you to see how the various semantic domains are related for other analyses than inter-coder agreement. For example, what kind of emotions are mentioned in relation to which actor and which strategies are used to deal with them. For this you can use the code co-occurrence tools. See Code Co-Occurrence Tools.
An important requirement for inter-coder agreement analysis is the independence of the coders. Thus, the person who has developed the code system cannot be one of the coders whose coding goes into the ICA analysis. In addition to the principal investigator who develops the code system, two or more persons are needed who apply the codes.
The coders will receive a project bundle file from the principle investigator (see Project Setup). It is recommended to provide the coders with the following instruction:
- When importing the project that you receive from the principal investigator, add your name or initials to the project name.
- After opening the project, double check your user name.
- Apply all codes of a semantic domain in a mutual exclusive manner.
- If fitting you can apply codes from multiple semantic domains to the same or overlapping quotations.
- Do not make any changes to the codes - do not alter the code definition, do not change the code name, or the code color.
- If you do not understand a code definition, or find a code label not fitting, create a memo and name it 'Comments from coder name'. Write all of your comments, issues you find and ideas you have into this memo.
- Do not consult with other coders. This is important for your coding to remain unbiased. The data must be coded by all coders independently.
- Once you are done coding, create a project bundle file and send it back to the principal investigator.