Scientific research seeks to explain and understand natural phenomena using empirical evidence. While visualization is often used in the exploration and presentation of that evidence, this panel will discuss improving the process and quality of quantitative visualization research itself.
The methods and practice of empirical research vary widely in the interdisciplinary field of visualization, which can sometimes lead to experimental results that are difficult to interpret. Recent efforts to reduce the use of Null Hypothesis Significance Testing has helped bring about more consistent and less dichotomous statistical reporting. However, statistical reporting is not the only facet of empirical research. How can we improve other components, including choosing questions, experiment design, analyses, and drawing conclusions? All are critical, and they must be consistent with each other to produce validly supported findings. Furthermore, each stage must be transparently communicated to be interpretable by other researchers. The goal is a body of reliable explanations which can be built upon to progress our understanding of how we perceive and think about visual information. Our panel will discuss the following questions about how to bring the field closer to that goal:
- How can we improve empirical thinking in visualization research?
- What questions are best suited for empirical research?
- How can perception and cognition help answer those questions?
- How can we improve reporting and communication of experiments and analyses?
- How can we reduce the effort needed for replication and increase the frequency of its occurrence?
- What statistical practices can improve the reliability of conclusions?
- Does the current reviewing process best support submissions with experiments?
The questions asked during the panel are available here
Pierre Dragicevic - Moderator (slides)
Pierre Dragicevic is a permanent research scientist at Inria since 2007, and studies information visualization (infovis) and human-computer interaction. He is interested in reforming statistical practice with a focus on replacing dichotomous testing with estimation thinking. He gives regular talks (e.g., at the BELIV 2014 biannual workshop) and publishes papers (alt.CHI '14, Chapter in upcoming Springer Book on Modern Statistical Methods for HCI) on the topic. He also maintains a Web page with reading material: www.aviz.fr/badstats
Steve Haroz - Clearly Stating Research Goals (slides)
Steve Haroz is a postdoctoral research fellow in the Psychology Department at Northwestern University. He researches how the brain perceives and understands visually displayed information, and he has experience with the experiment designs and statistics used in vision science and cognitive psychology. Steve also maintains a list of InfoVis publications which include statistically analyzed quantitative experiments: steveh.co/experiments
Improving empirical practices in visualization research requires clearly differentiating between two goals: (1) to scientifically explain perceptual or cognitive processes employed with visually presented data, and (2) to measure and optimize user performance in a specific context.
Scientific experiments that explain perceptual or cognitive processes yield explanations that can generalize broadly. They require careful isolation of components to determine how or why a phenomenon works, and they should have clear hypotheses that interpret results. For example, our understanding of opponent processing allows us to predict misperceptions in size or luminance. Also, understanding attentional limits can predict how visual complexity can impact performance. However, understanding the interactions between these low-level effects and scene-wide context often require an exponential number of additional theories and experiments. It can therefore be extremely challenging to use a simplified explanation to predict performance in a scenario that is more visually complex. In turn, the applicability of scientific explanations to visualization design is often limited.
However, when improving user performance for a specific context is the goal, a user study can provide easily interpretable and directly applicable results. The producer of a commercial security visualization package or a stock analysis platform aims to compare designs rather than understand neural mechanisms. Unfortunately from a research standpoint, it’s often unclear whether the results of such user studies generalize to any scenario without identical designs and datasets. Without an explanation for why an effect occurs, there is rarely an indication of what and how much can change while maintaining the benefits of a particular design.
Choosing between these two goals has substantial implications on how research is conducted and the generalizability vs. applicability of the end result. It is therefore imperative that any work with experiments or user studies states its goal clearly.
Ronald A. Rensink - Visualization Research and the Human Mind (slides)
Ronald A. Rensink (website & email) is an Associate Professor in the departments of Computer Science and Psychology at the University of British Columbia (UBC). His research interests include visual perception, information visualization and visual analytics. He obtained a PhD in Computer Science from the University of British Columbia (UBC) in 1992, followed by a postdoc in Psychology at Harvard University, and several years working as a research scientist for Nissan. He is currently part of the UBC Cognitive Systems Program, an interdisciplinary program combining Computer Science, Linguistics, Philosophy, and Psychology. He is also currently UBC Director of the Vancouver Institute for Visual Analytics (VIVA).
Two kinds of connections are proposed between research in visualization and what is known about human perception and cognition. The first involves issues in vision science. It is suggested that some knowledge of this area of research would be valuable, not only to help provide explanations of why particular visualization designs do (or do not) work, but also to provide guidelines for the design of systems that can work more effectively with humans, and perhaps even to provide new research directions. In addition, there is often considerable value in knowing ways of measuring performance based on visual displays, since many of these can also be applied to visualization systems.
The second kind of connection arises from having a sense of the strengths and weaknesses of human cognition, so that researchers and reviewers can avoid fooling themselves. For example, people need to step back from personal biases and consider other options, and make sure that they are not simply confirming their original biases. A related skill is to be open to negative feedback, so that people can evolve their viewpoints (and projects) in a way that is more likely to be productive. This may also be more conducive to effective interactions in a group, resulting in greater "group intelligence".
Jessica Hullman - Communicating Methods and Results (slides)
Jessica Hullman is an assistant professor in the Information School at the University of Washington and member of the Interactive Data Lab in Computer Science & Engineering. Her research interests focus on scientific and data communication aimed at non-expert audiences, with a goal of developing tools can help people reason more effectively with data using interactive visual interfaces. She frequently writes and presents on the topic of communicating uncertainty, and has experience in the design of experiments, statistical reasoning, and graphical comprehension from a cognitive psychology perspective. More details can be found on her website.
Empirical research is a conversation. Groups of researchers across disciplines and subfields draw on one another’s results (e.g., InfoVis and perceptual psychology), researchers interpret and publish their findings for discussion and iteration by others in their field, and each individual researcher must communicate honestly with him or herself to avoid to biasing empirical work toward desired, but not necessarily supported, conclusions. When communication around empirical research is not clear at any of these levels, evaluations of that work, and the subsequent research that builds on those findings, can be subject to false and/or superficial interpretations that threaten the validity of knowledge. I will describe communication challenges spanning different levels of the "research conversation" around empirical work in InfoVis. For example, researchers may overlook the uncertainty or ambiguity in the findings of prior work that they use to motivate their own hypotheses; reviewers may not thoroughly understand a reported experimental design, leading them to accept claims based on overinterpreted evidence; or researchers may fail to acknowledge the degree to which analysis procedures are constructed ad-hoc in an effort to confirm their hypothesis. I will offer ideas for discussion around how revising practices at different points in the pipeline of empirical research—from formulation of hypotheses, to consulting of the literature, to experimental design, to analysis and presentation of results, to review by the community—can help enforce clear communication.
Matt Kay - Usable Transparent Statistics (slides)
Matthew Kay is an Assistant Professor at the University of Michigan School of Information (starting Fall 2016). He studies the design of user-facing uncertainty in everyday sensing and prediction, such as personal informatics systems for health and applications for real-time transit prediction. He considers scientific communication a particular form of communicating uncertainty, and believes the statistical tools and techniques most common to the field (frequentist NHST) generate difficult-to-understand representations of uncertainty. He has published work advocating for a shift to the use of Bayesian estimation in VIS (InfoVis 2015) and HCI (CHI 2016). His website is: www.mjskay.com.
Rather than to persuade, the purpose of statistical communication should be to advance the state of our knowledge about the world. We should acknowledge the uncertainty in our evidence. We should build on the results of previous work. We should provide sufficient details in our work to allow others to build on our results. At the technical level, this perspective implies particular practices: estimation with uncertainty rather than binary hypothesis testing, Bayesian approaches or meta-analysis to combine estimates across studies, data and code sharing to make replication and reproduction possible. However, having technical solutions and a list of "should"s does not a solution make; shifting an entire field is complex and difficult, and yelling at researchers to do better when their tools do not support them amounts to blaming the users. How do we do better? How can we shift practice in the field? Besides sharing my perspective on where I think we should be going—particularly advocating for Bayesian estimation as a way to combine results across studies while acknowledging our uncertainty, and as a way to help designers and engineers of novel technology work symbiotically with empirical researchers—I will also share some insights from the CHI 2016 SIG on transparent statistical communication that may be applicable to VIS.