Bridging Objective and Subjective Evaluations in Data Visualization: A Crossover Experiment

One of the problems affecting evaluation in the design and adoption of HCI technology is that neither objective nor subjective measures are sufficient when taken alone or individually. This paper proposes a crossover approach, making sense of objective and subjective evaluation methods by hypothesizing them as constitutive of each other’s explanation. Objective image features borrowed from image processing may explain or being explained in terms of validated qualitative items for infographics value-in-use and qualitative labelling from users’ interaction. These methods are all applied to the evaluation of a small set of Data Vizualizations (Data Viz from now on). Image features are computed first, in order to provide a varied-features Data Viz selection from researchers; the subjective part of the evaluation is accomplished by the 98 participants of an experiment, who interacted with pairs of Data Viz by executing a task, then using the validated items of the Infographics-Value (IGV) short scale, and adding free qualitative comments. Crossing over these dimensions shows that: a high feature congestion in a Data Viz can hinder its perceived intuitiveness and clarity; a poorly distributed saliency may impact intuitiveness and clarity too; a high colorfulness may influence the perceived beauty; both saliency and colorfulness may impact on the perceived usefulness, informativity, and beauty. Furthermore, colorfulness can improve or worsen the perceived overall quality of design and quality of interaction when used and combined with feature congestion; and saliency may improve or worsen the perceived beauty when interacting with colorfulness. These results show how objective and subjective evaluations may be exploited as each other’s explanations for improving the evaluation process during both design and user experience with Data Viz. Based on this experiment, the importance of crossing-over quantitative and qualitative Data Viz evaluation is argued, and motivations to the exploitation of a combination of approaches instead of the application of one approach alone are supported. This contribution intends to lead towards a holistic Data Viz quality assessment method, able to provide a virtuous cycle enforcing both quantitative and qualitative approaches during all the phases of a Data Viz evaluation life.