Imagine you’re reviewing a submission. It has an experiment comparing how quickly subjects can get a correct answer using one of two visualization techniques. When measuring speed, it’s important to keep accuracy high. So whenever a subject would get an incorrect answer, the researchers would hammer a sharpened piece of bamboo under the subject’s fingernail. The results made strong advancements to our understanding of how people can extract information from charts. How would you respond to this submission as a reviewer?

Ethical compensation

Earlier today, I was on a panel that discussed ethical payment for study participants. Towards the end, the topic came up about what would happen if a reviewer comments that payment is unacceptably low. One panelist noted that that when IEEE VIS reviewers have raised ethical concerns about a submission poorly paying participants, the chairs dismissed those concerns because there is no explicitly stated rule about subject payment. (Edit: the panelist clarified that this was for a different ethical concert. But the premise still holds.) In fact, there is no rule in the IEEE VIS submission guidelines about human-subjects ethics at all.

This approach argues that all reasons for rejecting a paper must be enumerated in the submission and reviewing guidelines. In the few hours since the panel, I’ve been unable to shake how completely I reject this view as both unethical and impractical. A journal making use of unethical labor or unethical empirical research by publishing the submission is, itself, unethical. It is a second-degree exploitation. And practically, it is impossible for submission guidelines to preconceive every possible source of scientific invalidity or type of unethical behavior.

Nevertheless, it’s disturbing that the IEEE VIS submission guidelines make no mention of human-subjects ethics.

Bamboo under finger nails: Ethics beyond compensation

Going back to that first paragraph, what would you do if you were reviewing a submission and included some sort of unethical behavior. Maybe its poor payment. Maybe consent wasn’t obtained. Maybe Institutional Review Board (IRB) approval wasn’t obtained. Maybe there’s reason to question an institution’s IRB approval process. Maybe subjects were treated with horrific cruelty. 

I’m normally reticent to take a slippery slope argument. But in the case of human-subjects research, the Nuremberg code was created precisely because people have done horrifically cruel things to people in the name of research. It’s not a slippery slope if it has already happened.

In the case of IEEE VIS, there appears to be no mechanism for raising ethical concerns related to human-subjects, no matter how egregious they are. And if they are raised, they appear to be dismissed.

What about IEEE?

IEEE has a rule about human-subjects experiments in its operations manual:

8.1.1 – E) Authors of articles reporting on research involving human subjects or animals shall confirm upon submission of an article to the Editor* whether or not an approval was obtained from a relevant Review Board (or equivalent local/regional review). If such an approval was obtained, the original source and reference shall be provided to the Editor* at the time of submission and shall appear in the article.

It essentially says that an editor can request confirmation of ethical review, but that doesn’t include any mechanism for raising ethical concerns, such as whether the ethical approval was inappropriate.

Moreover, I’ve clashed with the IEEE VIS chairs in the past over their refusal to enforce rules that exist for IEEE but aren’t enumerated in the VIS submission guidelines. So it’s not clear if the chairs would even enforce a rule about requiring review board approval.

Don’t wait for enumeration

Just because IEEE VIS has chosen a negligent approach towards ethics, doesn’t mean that you have to accept unethical behavior. Whether it’s inappropriately low payment or putting subjects through some torturous squid game, ethical concerns supersede any restrictions placed on you by the limited enumeration of the submission and review guidelines.

Reviewers:

• Demand that any submission you review with human-subjects research explicitly state its ethical review approval.
• Note any ethical concerns you have regarding a submission. 
• If either of the above occur, give the submission the lowest possible score and explain your concerns as your review. Don’t provide constructive review for the submission until ethical concerns are addressed.

Program committee members and action editors:

• If a reviewer raises concerns and you agree, insist that the authors to address those concerns. Let them know that you are not comfortable forwarding other review comments until or unless the ethical concerns can be addressed.
• If the authors are unable to provide a satisfactory answer, consider contacting their institution’s review board to explain what you’ve found.

Chairs and editors in chief:

• Don’t dismiss ethical concerns by reviewers. It doesn’t mean you have to agree with or take action on every concern. But “we didn’t tell them that they have to be ethical” is a poor excuse for not taking human-subjects ethics seriously.
• Unambiguously enumerate ethical requirements such as noting review board approval in the paper.