Category Archives: Science

Guide to user performance evaluation at InfoVis 2016

Previous years: 2013, 2014, 2015

The goal of this guide is to highlight vis papers that demonstrate evidence of a user performance benefit. I used two criteria:

  1. The paper includes an experiment measuring user performance (e.g. accuracy or speed)
  2. Analysis of statistical differences determined whether results were reliable.

I did not discriminate beyond those two criteria. However, I am using a gold star to highlight one property that only a few papers have: a generalizable explanation for why the results occurred. You can read more about explanatory hypotheses here.

Continue reading

by 3 Comments

Guide to user performance evaluation at InfoVis 2015

The goal of this guide is to highlight vis papers that demonstrate evidence of a user performance benefit. I used two criteria:

  1. The paper includes an experiment measuring user performance (e.g. accuracy or speed)
  2. Analysis of statistical differences determined whether results were reliable.

I did not discriminate beyond those two criteria. However, I am using a gold star to highlight one property that only a few papers have: a generalizable explanation for why the results occurred. You can read more about explanatory hypotheses here.

Continue reading

by 1 Comment

3 Tips for Hiding Your Research Publications

As we all know, the most important part of publishing research is making sure that no one ever reads or cites it. After all, it’d be awful if anyone actually saw the end result of months or even years of your effort. So keep these tips in mind the next time you author a publication.
Continue reading

, by 1 Comment
citation not always needed (via XKCD)

Why I Don’t Write a “Related Work” Section

Imagine someone explaining a complex topic, like how to improve the fuel efficiency of a boat. But shortly after starting the explanation, they go off on a series of tangents about pretty boats they’ve seen, big boats, rubber ducks, submarines, and other transportation vehicles such as the new 787 by Boeing et al. Then they return to the explanation of boat efficiency without ever referencing why they brought up those strange tangents.

Tangents are confusing, and they hurt clarity. The related work section is often just a string of unrelated tangents, which is a waste of the reader’s time.

Now let me make something clear: I am not necessarily saying that papers should cite fewer sources. Instead, each citation should serve an obvious, specific purpose. And if that purpose is so tangential to the structure of your argument that you need to put it in what amounts to a citation dumping ground, then it isn’t needed.

What’s the purpose of a citation?

Continue reading

Posted in CHI, Science, Visualization by . 2 Comments

InfoVis 2014 – The methods papers

This year, 40% of InfoVis papers included an empirical evaluation. I made a list in my last post.

There were also a couple papers worth noting that described methods for evaluating visualizations. These papers can help bootstrap future evaluations, leading to a better understanding of when and why vis techniques are effective.

Learning Perceptual Kernels for Visualization Design – Çağatay Demiralp, Michael Bernstein, Jeffrey Heer pdf
A collection of methods are described to find the relative discriminability of feature values (e.g. colors or shapes). It also looks at finding the descriminability of combinations of visual features (e.g. colors and shapes). The paper validates its approach by determining the discriminability of size and showing which of their measures closely match the established Steven’s power law for size.

A Principled Way of Assessing Visualization Literacy – Jeremy Boy, Ronald Rensink, Enrico Bertini, Jean-Daniel Fekete pdf
This paper describes how to use Item Response Theory – a technique common in psychometrics and education literature – to assess a person’s “literacy” or skill with visualizations. I would have liked to have seen the approach validated (or at least compared) with some external factor like the person’s experience with visualization. Understandably, that can be tough to measure, but this method certainly shows promise for explaining individual differences in user performance.

by Leave a comment

Mysterious Origins of Hypotheses in Visualization and CHI

For years, I’ve noticed a strange practice in Visualization and CHI. When describing a study, many papers list a series of predictions and number them as H1, H2, H3… For example:

  • H1: Red graphs are better than blue graphs
  • H2: Participants will read vertical bar graphs more quickly than horizontal bar graphs

I have never seen this practice in any other field, and I was curious as to the origin.

Half Hypotheses

Although these statements are referred to as ‘hypotheses’, they’re not… at least, not completely. They are predictions. The distinction is subtle but important. Here’s the scientific definition of hypothesis according to The National Academy of Sciences:

A tentative explanation for an observation, phenomenon, or scientific problem that can be tested by further investigation…

The key word here is explanation. A hypothesis is not simply a guess about the result of an experiment. It is a proposed explanation that can predict the outcome of an experiment. A hypothesis has two components: (1) an explanation and (2) a prediction. A prediction simply isn’t useful on its own. If I flip a coin and correctly guess “heads”, it doesn’t tell me anything other than that I made a lucky guess. A hypothesis would be: the coin is unevenly weighted, so it is far more likely to land heads-up. It has an explanation (uneven weighting) that allows for a prediction (frequently landing heads-up).

The Origin of H1, H2, H3…

Besides the unusual use of the term “hypothesis”, where does the numbering style come from? It appears in many IEEE InfoVis and ACM CHI papers going back to at least 1996 (maybe earlier?). However, I’ve never seen it in psychology or social science journals. The best candidate I can think of for the origin of this numbering is a misunderstanding of null hypothesis testing, which can be best explained with an example. Here is a null hypothesis with two alternative hypotheses:

  • H0: Objects do not affect each other’s motion (null hypothesis)
  • H1: Objects attract each other, so a ball should fall towards the Earth
  • H2: Objects repel each other, so a ball should fly away from the Earth

Notice that the hypotheses are mutually exclusive, meaning only one can be true. In contrast, Vis/CHI-style hypotheses are each independent, and all or none of them can be true. I’m not sure how one came to be transformed into the other, but it’s my best guess for the origins.

Unclear

On top of my concerns about diction or utility, referring to statements by number hurts clarity. Repeatedly scrolling back and forth trying to remember “which one was H3 again?” makes reading frustrating and unnecessarily effortful. It’s a bad practice to label variables in code as var1 and var2. Why should it be better to refer to written concepts numerically? Let’s put an end to these numbered half-hypotheses in Vis and CHI.

Do you agree with this perspective and proposed origin? Can you find an example of this H numbering from before 1996? Or in another field?

, by 1 Comment