PROJECT TITLE :

An Empirical Study Into Annotator Agreement, Ground Truth Estimation, and Algorithm Evaluation

ABSTRACT:

Although agreement between the annotators who mark feature locations inside images has been studied within the past from a statistical viewpoint, little work has attempted to quantify the extent to that this phenomenon affects the evaluation of foreground–background segmentation algorithms. Many researchers utilize ground truth (GT) in experimentation and a lot of usually than not this GT comes from one annotator’s opinion. How will the difference in opinion affects an algorithm’s analysis? A methodology is applied to four image-processing problems to quantify the interannotator variance and to offer insight into the mechanisms behind agreement and the employment of GT. It is found that when detecting linear structures, annotator agreement is terribly low. The agreement during a structure’s position will be partially explained through basic image properties. Automatic segmentation algorithms are compared with annotator agreement and it's found that there is a clear relation between the 2. Several GT estimation ways are used to infer a range of algorithm performances. It is found that the rank of a detector is very dependent upon the method used to form the GT, which although STAPLE and LSML seem to represent the mean of the performance measured using individual annotations, when there are few annotations, or there is a massive variance in them, these estimates tend to degrade. Furthermore, one in every of the foremost commonly adopted combination methods—consensus voting—accentuates more obvious options, resulting in an overestimation of performance. It is concluded that in some information sets, it's not possible to confidently infer an algorithm ranking when evaluating upon one GT.