Every few months the same debate pops up in the open science circles in my Twitter feed, and we go round and round about whether people in general should sign their manuscript reviews. (The last time I saw it, it was an interesting Everything Hertz episode. Give it a listen, yeah?)
That conversation eventually evolves into whether certain people (early career researchers (ECRs), folks from minority groups, big names, people sending in positive/negative reviews, etc.) should be signing their reviews. I’ve been meaning to write about it myself, as I find it helps me sort out my own opinions. I like the idea of openness and transparency in reviews, but I also recognize that sometimes those things come with a cost. Do these costs outweigh the potential benefits? I don’t know, but I’m hoping to settle these feelings of ambivalence as I work through what, exactly, people claim the benefits and drawbacks are.
Benefits to Signing Peer Reviews
There are several reasons to advocate for folks signing their reviews. I’m sure I’m not going to cover all of them, but here are some of the ones I’ve heard about the most:
It’s the Right Thing to Do
Perhaps at its simplest, people should sign their reviews because it’s what they said. It would be reasonable to expect people to be willing to stand behind their words, regardless of whether the review is positive, negative, or as is usually the case, some mixture of the two. Maybe making people sign their reviews would prevent people from making lazy comments or trying to push unrelated literature they’ve authored into every paper they review. At the very least, it might prevent some grandstanding and rude comments, because, hey, that’s your name associated with those comments.
Get the Credit You Deserve
One compelling argument for open peer review is that you can get credit for your hard work. Even shorter reviews take a long time, and it can be difficult to confirm exactly (1) how many reviews you’re doing and (2) whether what you’re doing is even helpful. Even on a site like Publons, which is specifically used to track people’s contributions to peer review and publishing, it can be difficult for people to see exactly how much work you put into your review. If you sign your review, though, people can know your work. And hey…
It Could Open New Doors
Maybe you could get a new collaborator out of it! If someone appreciates your feedback on their paper, regardless of the ultimate decision, maybe that person will want to work with you on a project. Or maybe they’ll want to offer you an opportunity, like to review at their journal or serve on a committee. Maybe you could even get a postdoc or job offer!
In all honesty, I don’t know how often this happens, but maybe I don’t have enough faith in my own ability to write an amazing review with insight that truly impresses people. I do put a lot of effort into my reviews, but I find it a bit difficult to believe someone would scout me for something based on that effort.
It Holds People Accountable and Demystifies the Process
As some journals have moved to at the very least showing who has reviewed papers, it’s become easier to see the cracks in the traditional peer review system. In some of the more severe cases, it could help us when people have really crossed the line: for example, if someone reviews their own paper under someone else’s name. And if there are major problems for a paper, it can help us understand exactly what went wrong, and where.
Signing peer reviews, or perhaps going further and publishing them, can also demystify the process more generally. I can see an educational benefit for undergraduate and graduate students, who could benefit from seeing the process up close and maybe avoid some common problems. Something worth considering.
Drawbacks to Signing Peer Reviews
Just as there are many potential benefits to signing peer reviews, I also hear about a number of drawbacks. These are the ones I hear about most:
Retaliation
Sometimes I get really annoyed when I read feedback from reviewers, like the time a reviewer asked me for an “Instrumentation” section when I clearly had one, it was just called something else. It didn’t happen in that case, but sometimes reviews on my papers result in them getting rejected from the journal. That… stings. What do I do when that happens? I stew for a few days, maybe put the paper and its reviews aside for a bit, and then come back to it when it doesn’t hurt as much and I can clearly decide whether the paper is worth editing and sending off to another journal or scrapping entirely.
Other people don’t handle criticism–especially rejection–nearly as well. I know that from firsthand experience. Not from reviewing journal articles, specifically, but from criticizing people who ranked above me, including one particularly bad principal investigator. The retaliation I faced from him–and even other people in my department, though in more subtle ways–disrupted my ability to do the work I needed to do and might have affected some of my post-graduate-school opportunities.
So why don’t we have these conversations out in the open? That would stop the harassment, right? Who wouldn’t be embarrassed of having their colleagues find out how much of a baby they are about receiving negative feedback? So yeah, having people sign reviews could help us hold people accountable for their behavior. Besides, it doesn’t happen that often, right?
The thing that makes the linked story even sadder, in my opinion, is that other people, primarily women (that I saw), had spoken out about that harasser before, to crickets. No one offered them the support Eiko received, and even after his post, their experiences seemed secondary to his. This isn’t a slight against Eiko; he’s amazing and if I’m remembering correctly, he drew attention to this himself. It’s more of a comment on the way we often treat (read: ignore) those who have been harassed, particularly if they come from certain groups. This isn’t an issue with one or two bad actors, either; a quick search on harassment on a site like The Chronicle of Higher Education will provide a number of articles about professors and administrators violating harassment policies and the administrations that ignore or even cover up said employees’ behavior. That’s just the public stuff; there’s a lot that doesn’t get news coverage. So why open yourself up to the possibility that the person whose work you’re reviewing is, at best, a jerk who sends you an awful email threatening to ruin your reputation in the field, or at worst, someone who actually tries to do that?
Pressure to Reciprocate
Okay, so maybe don’t sign negative reviews. Just sign the nice ones! You can still get the benefit of name recognition and signaling you’re a fan of open science without dealing with someone who maybe doesn’t have good coping skills. Aside from feeling disingenuous, there are some concerns that maybe you (the reviewer) expect the author(s) to help you sometime, especially if you both research very similar things.
I don’t know how big of a problem this is, though I suppose I can see a situation where that could arise. I’ve received a couple of signed reviews in my limited time submitting papers to journals. I remember one in particular, from a big name in the field, who said a paper I was on was one of their top three easiest papers for them to review and a perfect paper for the journal we’d submitted to. I was over the moon given who it was, but I never felt like they expected me to say yes one day to one of their papers. It’s entirely possible I didn’t feel that way because of the sheer difference in status between us (their h-index is approximately infinitely larger than mine) or my status on the paper (I was not first author). My experience submitting to journals is limited, so this could be a bigger problem than I am anticipating.
The Verdict: Sign Peer Reviews?
Ultimately, I fall on the side of not publishing my name whenever I review manuscripts. My experiences within academia have led me to believe we still aren’t that good at standing up to our colleagues when they, at best, act inappropriately, and at worst, actively harass and/or retaliate against other scientists. This is true even within organizations and fields that claim to value equity and diversity. Given that the number of papers someone publishes (and what journals they publish in) still affect things like hiring, promotion, and what snazzy awards you get, and probably will still affect those things for quite some time, there’s still pretty good incentive to not be your best self when you get a rejection letter.
Beyond that, though, we do often judge the message in part based on the messenger, and being an early career researcher, I can easily envision a scenario where someone assumes I don’t know what I’m talking about because I just got my PhD a few years ago (or because I’m a woman, or I’m not at an R1 institution, or I’m not a ________ psychologist, or…). In the event I am reviewing someone who “ranks” lower than me–an increasing inevitability as I move along in my career–I don’t want someone to hesitate to defend their work against my criticisms because of who I am. If I want my message to be judged based on its merits, it seems to me that I shouldn’t give people reasons–other than what I’ve written–to dismiss what I have to say.
I also think maybe there are other things about peer review we need to fix before we start worrying about whether people sign their reviews. I’m reminded of the PNAS study that found no agreement among reviewers looking at the same application. Why add more potential sources of bias to a system that already can’t agree on what it means to have a quality paper (or grant proposal)?
In other words, I’m not against the idea of signing reviews, and if you want to do it, go ahead. The calculus might look very different for you, especially if you aren’t an early career researcher. I do think we underestimate the influence of having names attached to reviews. We tend to think we’re above such influences. Spoiler alert: We’re not.
I’m open to changing my mind, and maybe I will, as the culture within academia shifts. It’ll require journals creating codes of conduct (which some are) and editors being more mindful of the content of reviews over the simple “accept/reject” ratio (which some are). Maybe we can follow the lead of some journals, where they list reviewers’ names but not individual comments, and only for papers that get published. Slightly problematic when there’s only one reviewer, but hopefully that’s fairly rare anyway. Something like this could deal with issues like the person who was making up reviewers.
Right now, the potential drawbacks weigh heavier on my mind than the potential benefits, but that calculus may eventually change. Maybe journals will do a better job of filtering out terrible reviews (and their associated reviewers), and maybe academics dedicated to improving research practices will do better at realizing harassment harms science and actually take steps to prevent it from happening to our peers.
Until then, call me Reviewer #1.