This is part 3 of a multi-part post summarizing open practices in visualization research for 2019, as displayed on Open Access Vis. Research openness can either rely on policy or individual behavior. In this part, I’ll look at the individuals. Who in the visualization community is consistently sharing the most research? And who is not?

Related posts: 2017 overview, 2018 overview, 2019 part 1 – Updates and Papers, 2019 part 2 – Research Practices

Whose papers are open?

Many authors are sharing most or even all of their papers on open repositories, which is fantastic progress. But many are not, despite encouragement after acceptance. Easier options, better training, and formal policies will likely be necessary for a field-wide change in behavior.

It’s certainly possible that I’ve missed some papers and artifacts, so please let me know if I’ve made any mistakes.

Whose research artifacts are open?

While paper sharing has substantially increased, open research artifacts are still uncommon. Here, I’m again using the generous “Is anything open?” measure from part 2.

VIS Leadership

It is disappointing to see members of nearly every VIS organizing group (including the Visualization Executive Committee, steering committees, and paper chairs) among those who do not make their research open. The VEC recently released a statement which claims to strongly support an encouragement approach to open research practices. I was curious whether that support translates into behavior, so I compared papers that have a VEC member as a coauthor with those that don’t.

For both paper sharing and availability of research artifacts, papers with VEC members are slightly behind the rest of the field. With only half of VEC-coauthored papers shared and only 1 out of 16 sharing any research artifact, it seems that setting a good example for transparency is not a part of an encouragement approach.

Notable Exemplars

Niklas Elmqvist was a coauthor on the most papers with open research practices. All three papers are on a persistent repository and have at least one research artifact shared. The work of Niklas and his coauthors is discoverable and transparent, making it more credible and reusable.

• Common Fate for Animated Transitions in Visualization
  Amira Chalbi, Jacob Ritchie, Deokgun Park, Jungu Choi, Nicolas Roussel, Niklas Elmqvist, Fanny Chevalier
        
• There Is No Spoon: Evaluating Performance, Space Use, and Presence with Expert Domain Users in Immersive Analytics
  Andrea Batch, Andrew Cunningham, Maxime Cordeil, Niklas Elmqvist, Tim Dwyer, Bruce H Thomas, Kim Marriott
        
• The Perceptual Proxies of Visual Comparison
  Nicole Jardine, Brian David Ondov, Niklas Elmqvist, Steven Franconeri
        

For the first time at VIS, a paper shares all measured research artifacts. Better yet, two of them do! You can find their preregistrations, data collection material, raw empirical data, and analysis/computational material in the papers.

• Evaluating Gradient Perception in Color-coded Scalar Fields
  Khairi Reda, Michael E. Papka
        
• Evidence for Area as the Primary Visual Cue in Pie Charts
  Robert Kosara
        

Moving VIS forward

What you can do:

• Practice! Try putting one artifact of your research on a repository like OSF. See tips and guides for data management and various FAQs on open practices in visualization research.
• Learn more about preregistration for empirical research to make the timing of your decisions more transparent.
• Account for openness or a lack thereof in your reviewing. Consider signing the Peer Reviewers’ Openness Initiative.
• Set the norm. Put your research on a persistent open repository like OSF, and encourage others to do the same.

Visualization research owes its existence to others sharing the data they’ve collected. For visualization research to not be similarly transparent is selfish and hypocritical. Moreover, preventing others from scrutinizing the research raises questions about the field’s credibility. A norm of research openness will greatly benefit those doing credible research and could benefit other fields just as visualization has benefited from the openness of others.

Transparency doesn’t guarantee credibility – it guarantees you’ll get the credibility you deserve.
– Simine Vazire

All code and data for this series of posts is available at https://osf.io/4njyf