{"id":816,"date":"2017-10-03T17:21:37","date_gmt":"2017-10-03T22:21:37","guid":{"rendered":"http:\/\/steveharoz.com\/blog\/?p=816"},"modified":"2017-10-03T18:44:15","modified_gmt":"2017-10-03T23:44:15","slug":"guide-to-user-performance-evaluation-at-infovis-2017","status":"publish","type":"post","link":"http:\/\/steveharoz.com\/blog\/2017\/guide-to-user-performance-evaluation-at-infovis-2017\/","title":{"rendered":"Guide to user performance evaluation at InfoVis 2017"},"content":{"rendered":"

Previous years: 2013<\/a>, 2014<\/a>, 2015<\/a>, 2016<\/a><\/p>\n

The goal of this guide is to highlight vis papers that demonstrate\u00a0evidence of a user performance benefit. I used two criteria:<\/p>\n

    \n
  1. The paper includes an\u00a0experiment measuring user performance (e.g. accuracy or speed)<\/li>\n
  2. Analysis of statistical differences determined whether\u00a0results were reliable.<\/li>\n<\/ol>\n

    I\u00a0did not discriminate beyond those two criteria. However, I am using a gold star \u2605<\/span> to highlight one property\u00a0that only a few papers have: a generalizable explanation for why the results occurred. You can read more about explanatory hypotheses here<\/a>.<\/p>\n


    \nActive Reading of Visualizations<\/strong> – Jagoda Walny, Samuel Huron, Charles Perin, Tiffany Wun, Richard Pusch, and Sheelagh Carpendale
    \n    pdf<\/a><\/p>\n

    Assessing the Graphical Perception of Time and Speed on 2D + Time Trajectories<\/strong> – Charles Perin, Tiffany Wun, Richard Pusch, and Sheelagh Carpendale
    \n    pdf<\/a><\/p>\n

    Explanatory<\/a>Blinded with Science or Informed by Charts? A Replication Study<\/span><\/strong> – Pierre Dragicevic and Yvonne Jansen
    \n    pdf<\/a><\/p>\n

    Conceptual and Methodological Issues in Evaluating Multidimensional Visualizations for Decision Support<\/strong> – Evanthia Dimara, Anastasia Bezerianos, and Pierre Dragicevic
    \n    pdf<\/a><\/p>\n

    Explanatory<\/a>Data Through Others\u2019 Eyes: The Impact of Visualizing Others\u2019 Expectations on Visualization Interpretation<\/span><\/strong> – Yea-Seul Kim, Katharina Reinecke, and Jessica Hullman
    \n    pdf<\/a><\/p>\n

    EdWordle: Consistency-preserving Word Cloud Editing<\/strong> – Yunhai Wang, Xiaowei Chu, Chen Bao, Lifeng Zhu, Oliver Deussen, Baoquan Chen, and Michael Sedlmair
    \n    pdf<\/a><\/p>\n

    The Hologram in My Hand: How Effective is Interactive Exploration of 3D Visualizations in Immersive Tangible Augmented Reality?<\/strong>\u00a0– Benjamin Bach, Ronell Sicat, Maxime Cordeil, Johanna Beyer, and Hanspeter Pfister
    \n    pdf<\/a><\/p>\n

    Imagining Replications: Graphical Prediction & Discrete Visualizations Improve Recall & Estimation of Effect Uncertainty<\/strong>\u00a0– Jessica Hullman, Matthew Kay, Yea-Seul Kim, and Samana Shrestha
    \n    pdf<\/a><\/p>\n

    Modeling Color Difference for Visualization Design<\/strong>\u00a0– Danielle Albers Szafir
    \n    pdf<\/a><\/p>\n

    Open vs Closed Shapes: New Perceptual Categories?<\/strong>\u00a0– David Burlinson, Kalpathi Subramanian, and Paula Goolkasian
    \n[This publication is hidden]<\/span><\/p>\n

    Explanatory<\/a>Taking Word Clouds Apart: An Empirical Investigation of the Design Space for Keyword Summaries<\/span><\/strong>\u00a0
    \nCristian Felix, Enrico Bertini, and Steven Franconeri
    \n    pdf<\/a><\/p>\n

    Visualizing Nonlinear Narratives with Story Curves<\/strong>
    \nNam Wook Kim, Benjamin Bach, Hyejin Im, Sasha Schriber, Markus Gross, and Hanspeter Pfister
    \n    pdf<\/a><\/p>\n

    Evaluating Cartogram Effectiveness<\/strong>
    \nSabrina Nusrat, Muhammad Jawaherul Alam, Stephen Kobourov
    \n    pdf<\/a><\/p>\n

    Evaluating Interactive Graphical Encodings for Data Visualization<\/strong>
    \nBahador Saket, Arjun Srinivasan, Eric D. Ragan, Alex Endert
    \n    pdf<\/a><\/p>\n

    Explanatory<\/a>Perceptual Biases in Font Size as a Data Encoding<\/span><\/strong>
    \nEric Carlson Alexander, Chih-Ching Chang, Mariana Shimabukuro, Steve Franconeri, Christopher Collins, Michael Gleicher
    \n    pdf<\/a><\/p>\n

    A flat trend<\/h2>\n

    31%<\/strong> of InfoVis conference papers measured user performance. Despite the proportion of papers with experiments rising, there has been little\u00a0change in the proportion measuring performance.<\/p>\n

    Here’s\u00a0the Aggresti-Coull binomial 84% CI, so each proportion can be compared.<\/p>\n

    \"2016\"<\/a><\/p>\n

    Little\u00a0Generalization<\/h2>\n

    There is still a very low proportion of papers with an explanatory hypothesis that can inform\u00a0generalizability. This assessment is tough to assign, but I try to be very generous with this assessment. Very few papers attempt to explain why or if the results are applicable outside of the specific conditions of the study. Also, there are still a lot of guesses presented as hypotheses.<\/p>\n

    Obviously, please let me know if you find a mistake or think I missed something. Also, please hassle any authors who didn\u2019t make their pdf publicly available.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

    Previous years: 2013, 2014, 2015, 2016 The goal of this guide is to highlight vis papers that demonstrate\u00a0evidence of a user performance benefit. I used two criteria: The paper includes an\u00a0experiment measuring user performance (e.g. accuracy or speed) Analysis of statistical differences determined whether\u00a0results were reliable. I\u00a0did not discriminate beyond those two criteria. However, I […]<\/p>\n","protected":false},"author":1,"featured_media":825,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":3,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":""},"categories":[4],"tags":[15],"class_list":["post-816","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-science","tag-infovis"],"_links":{"self":[{"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/posts\/816","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/comments?post=816"}],"version-history":[{"count":10,"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/posts\/816\/revisions"}],"predecessor-version":[{"id":828,"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/posts\/816\/revisions\/828"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/media\/825"}],"wp:attachment":[{"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/media?parent=816"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/categories?post=816"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/steveharoz.com\/blog\/wp-json\/wp\/v2\/tags?post=816"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}