Inter-rater reliability of query/probe-based techniques for measuring situation awareness.
UNLABELLED
Query- or probe-based situation awareness (SA) measures sometimes rely on process experts to evaluate operator actions and system states when used in representative settings. This introduces variability of human judgement into the measurements that require inter-rater reliability assessment. However, the literature neglects inter-rater reliability of query/probe-based SA measures. We recruited process experts to provide reference keys to SA queries in trials of a full-scope nuclear power plant simulator experiment to investigate the inter-rater reliability of a query-based SA measure. The query-based SA measure demonstrated only 'moderate' inter-rater reliability even though the queries were seemingly direct. The level of agreement was significantly different across pairs of experts who had different levels of exposure to the experiment. The results caution that inter-rater reliability of query/probe-based techniques for measuring SA cannot be assumed in representative settings. Knowledge about the experiment as well as the domain is critical to forming reliable expert judgements.


PRACTITIONER SUMMARY
When the responses of domain experts are treated as the correct answers to the queries or probes of SA measures used in representative or industrial settings, practitioners should take caution in assuming (or otherwise assess) inter-rater reliability of the situation awareness measures.
['Lau|Nathan|N|', 'Jamieson|Greg A|GA|', 'Skraaning|Gyrd|G|']
D001364:Awareness D003198:Computer Simulation D036262:Empirical Research D006801:Humans D055848:Nuclear Power Plants D015588:Observer Variation D011594:Psychometrics D015203:Reproducibility of Results D013647:Task Performance and Analysis
nuclear experiments full-scope simulator performance assessment process control ratings subject-matter-experts