{"id":652,"date":"2016-10-23T12:48:21","date_gmt":"2016-10-23T17:48:21","guid":{"rendered":"http:\/\/steveharoz.com\/blog\/?p=652"},"modified":"2016-10-27T23:56:59","modified_gmt":"2016-10-28T04:56:59","slug":"guide-to-user-performance-evaluation-at-infovis-2016","status":"publish","type":"post","link":"http:\/\/steveharoz.com\/blog\/2016\/guide-to-user-performance-evaluation-at-infovis-2016\/","title":{"rendered":"Guide to user performance evaluation at InfoVis 2016"},"content":{"rendered":"

<\/p>\n

Previous years: 2013<\/a>, 2014<\/a>, 2015<\/a><\/p>\n

The goal of this guide is to highlight vis papers that demonstrate\u00a0evidence of a user performance benefit. I used two criteria:<\/p>\n

    \n
  1. The paper includes an\u00a0experiment measuring user performance (e.g. accuracy or speed)<\/li>\n
  2. Analysis of statistical differences determined whether\u00a0results were reliable.<\/li>\n<\/ol>\n

    I\u00a0did not discriminate beyond those two criteria. However, I am using a gold star \u2605<\/span> to highlight one property\u00a0that only a few papers have: a generalizable explanation for why the results occurred. You can read more about explanatory hypotheses here<\/a>.<\/p>\n


    \n    Explanatory<\/a>The Attraction Effect in Information Visualization<\/span><\/strong> – Evanthia Dimara, Anastasia Bezerianos, and Pierre Dragicevic.
    \n    pdf<\/a><\/p>\n

    Explanatory<\/a>Colorgorical: Creating discriminable and preferable color palettes for information visualization<\/span><\/strong> – Connor Gramazio, David Laidlaw, and Karen Schloss.
    \n    pdf<\/a><\/p>\n

    The Connected Scatterplot for Presenting Paired Time Series<\/strong> – Steve Haroz, Robert Kosara, Steven Franconeri.
    \n    pdf<\/a><\/p>\n

    Explanatory<\/a>Evaluating the Impact of Binning 2D Scalar Fields<\/span><\/strong> – Lace Padilla, P. Samuel Quinan, Miriah Meyer, and Sarah Creem-Regehr.
    \n    pdf<\/a><\/p>\n

    An Evaluation of Visual Search Support in Maps<\/strong> – Rudolf Netzel, Marcel Hlawatsch, Michael Burch, Sanjeev Balakrishnan, Hansj\u0094rg Schmauder, and Daniel Weiskopf.
    \n[This publication is hidden]<\/span><\/p>\n

    Immersive Collaborative Analysis of Network Connectivity: CAVE-style or Head-Mounted Display?<\/strong> – Maxime Cordeil, Tim Dwyer, Karsten Klein, Bireswar Laha, Kim Marriott, and Bruce H. Thomas.
    \n    pdf<\/a><\/p>\n

    Many-to-Many Geographically-Embedded Flow Visualisation: An Evaluation<\/strong> – Yalong Yang, Tim Dwyer, Sarah Goodwin, and Kim Marriott.
    \n    pdf<\/a><\/p>\n

    Map LineUps: effects of spatial structure on graphical inference<\/strong> – Roger Beecham, Jason Dykes, Wouter Meulemans, Aidan Slingsby, Cagatay Turkay, and Jo Wood.
    \n    pdf<\/a><\/p>\n

    Optimizing Hierarchical Visualizations with the Minimum Description Length Principle<\/strong> – Rafael Veras and Christopher Collins.
    \n    pdf<\/a><\/p>\n

    A Study of Layout, Rendering, and Interaction Methods for Immersive Graph Visualization<\/strong> – Oh-Hyun Kwon, Chris Muelder, Kyungwon Lee, Kwan-Liu Ma.
    \n    pdf<\/a><\/p>\n

    Towards Unambiguous Edge Bundling: Investigating Confluent Drawings for Network Visualization<\/strong> – Benjamin Bach, Nathalie Henry Riche, Christophe Hurter, Kim Marriott, and Tim Dwyer.
    \n    pdf<\/a><\/p>\n

    VLAT: Development of a Visualization Literacy Assessment Test<\/strong> – Sukwon Lee, Sung-Hee Kim, and Bum Chul Kwon.
    \n    pdf<\/a><\/p>\n

    A flat trend<\/h2>\n

    27%<\/strong> of InfoVis conference papers measured user performance – a 10% drop from last year. Overall, there has been little\u00a0change in the past four years.<\/p>\n

    Here’s\u00a0the Aggresti-Coull binomial 84% CI, so each proportion can be compared.<\/p>\n

    \"2016\"<\/a><\/p>\n

    The Journal\u00a0Articles<\/h2>\n

    Although TVCG appears to have taken a big dive, the total number of articles\u00a0is so low, that it could very well be noise. Note the large error ribbon for TVCG.<\/p>\n

    In the chart on the right, I collapsed the past four\u00a0years of data and recomputed the means and CIs. For TVCG, I’m only include papers presented in\u00a0the InfoVis track.\u00a0TVCG has more papers with performance evaluation, and it can’t simply be explained by random noise.<\/p>\n

    <\/h2>\n

    Little\u00a0Generalization<\/h2>\n

    There is still a very low proportion of papers with an explanatory hypothesis that can inform\u00a0generalizability. I try to be very generous with this assessment, but very few papers attempt to explain why or if the results are applicable outside of the specific conditions of the study. Also, there are still a lot of guesses presented as hypotheses.<\/p>\n

    Obviously, please let me know if you find a mistake or think I missed something. Also, please hassle any authors who didn\u2019t make their pdf publicly available.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

    Previous years: 2013, 2014, 2015 The goal of this guide is to highlight vis papers that demonstrate\u00a0evidence of a user performance benefit. I used two criteria: The paper includes an\u00a0experiment measuring user performance (e.g. accuracy or speed) Analysis of statistical differences determined whether\u00a0results were reliable. I\u00a0did not discriminate beyond those two criteria. However, I am […]<\/p>\n","protected":false},"author":1,"featured_media":665,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":3,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":""},"categories":[4,6],"tags":[15],"class_list":["post-652","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-science","category-visualization","tag-infovis"],"_links":{"self":[{"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/posts\/652","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/comments?post=652"}],"version-history":[{"count":23,"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/posts\/652\/revisions"}],"predecessor-version":[{"id":678,"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/posts\/652\/revisions\/678"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/media\/665"}],"wp:attachment":[{"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/media?parent=652"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/categories?post=652"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/tags?post=652"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}