Exploring Data Uncertainty through Speech, Text, and Visualization

In the realm of data-driven decision-making, effectively communicating the uncertainty present in data is critical. Data uncertainty refers to the range of potential outcomes, variability within a dataset, or possible errors in measurements or predictions. Understanding and communicating data uncertainty is crucial for making informed decisions, especially in sectors like finance, healthcare, and government/public health. Communicating this uncertainty can improve understanding and decision-making, but it also poses significant challenges, especially in ensuring accurate interpretation by non-experts.

This blog post describes recent research that explores how variations within speech, text, and visualization modes contribute to communicating uncertainty to users, examining each mode's affordances and limitations. Findings from two crowdsourced experiments indicate that visualization and text are most effective for rational decision-making, though text leads to lower confidence, while speech garners the highest trust despite sometimes leading to risky decisions.

Modes of Communicating Uncertainty

Visualization tools like error bars, confidence intervals, and density plots illustrate data variability and distribution. However, their effectiveness depends on the viewer's graphical literacy. Some visualization techniques can support data uncertainty better than others, such as bubble treemaps and ensemble visualizations representing a range of estimates.

Text can convey uncertainty through hedge words (e.g., "somewhat," "possibly"). Computational linguistics has developed techniques to identify and assess the effectiveness of these hedges in communication. While hedge words can decrease perceived credibility, they may enhance trust when used appropriately, such as in medical research descriptions.

Speech adds an acoustic dimension to uncertainty communication. Confidence in speech can be conveyed through faster speech rates, higher intensity, and fewer pauses. Acoustic cues can significantly influence listeners' perception of certainty, affecting decision-making processes.

Exploring the Modes

Understanding the nuances of how we communicate data uncertainty is a crucial yet complex challenge. This research delves into the intricate dynamics of how speech, text, and visualization modes influence decision-making. By systematically exploring these modes, the study aims to uncover the most effective strategies for conveying uncertain information, ultimately guiding users toward better-informed decisions. The research explored two primary questions:

1. Impact of Presentation Modes on Decision-Making: How does the way information is presented (speech, text, or visualization) affect decision-making with uncertain data?
2. Intensifying Uncertainty Indicators: How does decision-making change when the signs of uncertainty are increased within each mode of communication?

To address these questions, two crowdsourced experiments were conducted:

Experiment 1: Mode Comparison

This experiment set out to compare the effectiveness of speech, text, and visualization in conveying uncertain information and if differences even existed. Participants were presented with the same data in different modes, and their decision-making processes were analyzed to determine which mode best supported rational and confident choices. We also varied the perceived certainty of the representations to test a variety of stimuli, using “fuzzy” and “concrete” stimuli.  Fuzziness involves unclear boundaries that allow for multiple interpretations, while concreteness denotes specific and clear concepts with more precise meanings to minimize ambiguity. Both attributes are essential in communicating data uncertainty. Experiment 1 stimuli are shown in Table 1.

Experiment 2: Heightening Uncertainty

Building on the insights from the first experiment, the second experiment focused on the effects of intensifying uncertainty indicators within each mode. By manipulating variables such as speech rate, hedge words, and visual fuzziness, the study aimed to observe how these changes influenced participants' decision-making and trust in the information provided.

Table 2: Stimulus categories and examples for Experiment 2. Cells with a light gray background indicate stimuli that overlapped with Experiment 1 (E1). Only a sample of the text stimuli is provided for space considerations. Visual treatments shown here were applied to stimuli in the survey.

Takeaways

The research unveiled interesting insights into how different modes of communication affect decision-making under uncertainty. By comparing the efficacy of visual, textual, and speech-based presentations, we identified unique strengths and trade-offs for each mode. Here are the key findings from the two experiments:

Decision-Making Efficacy

Visualization: The Rational Choice. Visualizations emerged as the strongest mode for supporting rational decision-making. By providing clear and direct representations of data, visualizations help users make more informed and logical choices, making them a powerful tool for navigating uncertain information.

Text: Confidence Paradox. Textual communication also supports rational decision-making, but with a twist. While it helps users make informed decisions, it tends to lower their confidence in those decisions. The presence of hedge words and the lack of visual aids may contribute to this confidence gap, indicating a need for complementary modes to boost user assurance.

Speech: Trust vs. Risk. Speech communication stands out for its ability to garner high levels of trust from users. However, this trust comes with a downside—speech can sometimes lead to riskier decisions. The nuances of tone, pitch, and prosody may make the information feel more credible, but they can also obscure the uncertainty, leading users to take more significant risks.

Effect of Uncertainty Intensification

Stable Decision Quality. Interestingly, increasing the indicators of uncertainty within each mode did not significantly impact the quality of decision-making. Whether through more pronounced hedge words in text, varying speech rates, or fuzzier visualizations, the decision-making efficacy remained stable. This suggests that while these modes have different strengths, their ability to convey uncertainty effectively is not easily swayed by intensifying these signals.

Consistent Trust and Confidence. The trust and confidence findings remained consistent across different levels of uncertainty intensification. Visualization continued to support rational decisions, text remained a strong yet confidence-lowering mode, and speech maintained its high trust with potential for risky choices. These consistent results highlight the trade-offs and strengths of each mode, providing valuable insights for developing multimodal strategies for uncertainty communication.

OK, this is all cool. But what’s next?

Future research should explore the potential of multimodal representations to enhance the communication of data uncertainty. By combining visual, textual, and auditory elements, we can create a more robust and effective communication strategy. For instance, pairing visualizations with explanatory text can provide the clarity and confidence needed for rational decision-making, while also leveraging the trust elicited by speech. This integrated approach could balance the strengths and weaknesses of each mode, leading to more useful experiences. Additionally, exploring the development of interactive interfaces that allow users to switch between or simultaneously view different modes of information could enhance user experience. Conducting studies to understand user preferences for multimodal representations in various contexts, such as medical decisions, financial planning, or public policy, can further inform the design of effective communication strategies. This research is supported by findings from follow-up work, which demonstrated that there is no one-size-fits-all approach to uncertainty communication. By implementing and evaluating multimodal prototypes, the study highlighted the importance of user preferences and situational context in designing multimodal strategies.

Design Space Expansion

Expanding the design space involves exploring additional characteristics within each mode that could influence the perception and communication of uncertainty. This includes not only the content but also the presentation style and contextual factors that might affect user interpretation. Specific areas for expansion include examining how different font styles, sizes, and colors impact the readability and perceived certainty of textual information. Bold, italics, and color coding could emphasize certain aspects of the data, potentially improving comprehension.

For speech characteristics, investigating how variations in speaker attributes such as accent, tone, pitch, and speed affect trust and decision-making is essential. Options for these attributes continue to evolve as synthesized speech platforms expand. Understanding these nuances can help tailor spoken communications to different audiences and contexts. Similarly, exploring new visualization techniques and enhancements, such as animated transitions, or the use of color gradients to indicate varying levels of certainty, could provide deeper insights into effective uncertainty communication.

Accessibility

The research was tested with sighted users, but ensuring that uncertainty communication is accessible to all users is a critical aspect of future research. This involves designing modes of communication that cater to people with different abilities, ensuring that everyone can understand and use the information effectively. Strategies for accessibility include developing visualizations that are easily interpretable for individuals with color blindness, low vision, or other visual impairments. This could involve using high-contrast colors, alternative text descriptions, and customizable display settings.

Creating speech-based communications that consider the needs of people with hearing disabilities might involve providing transcripts, subtitles, or using clear and articulate speech synthesis. Designing textual and visual information to be easily understandable for individuals with cognitive impairments by simplifying language, avoiding jargon, and using clear and consistent layouts can enhance comprehension. Additionally, exploring multisensory approaches that combine auditory, visual, and tactile elements can cater to a wider range of abilities and preferences.

Implementing accessibility features requires engaging with diverse user groups to test and refine communication methods, ensuring they meet the needs of different audiences. Developing and adhering to accessibility guidelines and standards, such as the Web Content Accessibility Guidelines (WCAG), can ensure consistent and inclusive design practices. Educating designers, developers, and communicators about the importance of accessibility and how to implement inclusive design strategies effectively is also crucial for creating equitable data uncertainty communication artifacts.

Wrapping Up

Effectively communicating data uncertainty is essential for informed decision-making across various domains. This research highlights the strengths and limitations of speech, text, and visualization in conveying uncertainty. While visualization and text are effective for rational decisions, speech is highly trusted but can lead to risky choices. These findings have possible implications for future research into multimodal and accessible approaches to uncertainty communication.

This post summarizes the findings of the paper "From Delays to Densities: Exploring Data Uncertainty through Speech, Text, and Visualization," presented at the Eurographics Conference on Visualization (EuroVis) 2024. The research was conducted by Chase Stokes, Chelsea Sanker, Bridget Cogley, and Vidya Setlur.