Comments on ANSI/NISO Z39.106-2023 - Standard Terminology for Peer Review

  1. Janaynne Carvalho do Amaral

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

In July 2023, version 3.0 of Standard Terminology for Peer Review was published by the National Information Standards Organization. The terminology approaches four aspects of the peer review process: identity transparency, reviewer interacts with, review information published, and post-publication commenting. In this article, I comment on some gaps in the categories of identity transparency, reviewer interacts with, and reviewer information. To demonstrate these gaps, I will use some examples of open peer review models with public participation in journals from different disciplines. Furthermore, I would like to comment on how the information about identity transparency is presented and propose the inclusion of a category for the type of reviewers who may evaluate a paper. In short, version 3.0 of the NISO Peer Review Terminology is a great initiative to support journals and peer review, but not apply to all review models and it is intended.

Article activity feed

  1. PREreview

    This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/14618903.

    Major issues

    • I'd edit your background and conclusion to help the reader understand why the taxonomy is important for the scholarly communication ecosystem in order to motivate your suggestions in the beginning and drive home the impact at the end. I imagine there may be many readers who are tangentially familiar with conversations around peer review models, the NISO taxonomy, etc. but don't see that rhetoric as integral to their own work in scholarly communication, so anything you can do to connect this to the bigger picture would be helpful for getting a wider audience to engage meaningfully with your work.

    • A source of pushback to your paper would be exactly what you mention in the background: that the taxonomy was intended to cover many models of peer review in some level of detail but not every model in exhaustive detail. The taxonomy mentions that publishers should include more information as appropriate, so a response to your paper might be, "why does the taxonomy need to include these additional details if it'd only apply to a few unusual journals?" For that reason, it feels essential that you articulate why exactly the changes you suggest are key to include in the taxonomy and are not just oddities or minor comments without major impact on the usefulness of the taxonomy and its contribution to scholarly communication. To this end, I'd suggest adding a brief method section that describes your methodology for identifying journals that are not accurately described by the taxonomy and selecting the taxonomy criteria that represented a key gap versus oddities. Some analysis of how many journals would be impacted by your suggested changes or how they connect to broader trends in scholarly communication would also add some robustness to this potential rebuttal, though I'm not sure if the return on the time investment would be warranted or not. (That being said, perhaps there's already analysis on this out there that you could use!)

    Minor issues

    • In the conclusion, when you say, "From the side of the editors, I would encourage them to go beyond the use of terminology and make their peer review policies and reviewers' guidelines visible on their homepage, besides providing simple visual flows of their peer review process. Editors could even ask for feedback from their community about the accessibility of the information about the peer review process that they provide – is it too long? it is too short? Is it helpful? Is it easy to find on the journal's website?," I think this might be more pointed at publishers who own and control the webpages and that a common response would be that peer review policies and reviewer guidelines are already linked on journal homepages and that including those details on the homepage would make it cumbersome to navigate from a user experience perspective. Given that it's a recommendation, I'd move it to the body with your other recommendations and articulate why linking to editorial policies and reviewer guidelines in a traditional manner is insufficient. As a side note, I love the idea here to essentially encourage editors to peer review their peer review policy and its presentation :-)

    • I'd start off the background by describing what your paper is setting out to do before describing how the taxonomy came to be. The taxonomy's genesis is useful for a reader unfamiliar with it but not core to your argument, so I'd just change the emphasis in your background so that the focus remains on the meat of your suggestions.

    • Similarly, I think tightening up the structure of your argument and using the background, headings, and conclusion to more clearly walk the reader through it would make the paper more impactful. I know this is an expositional suggestion and am thus hesitant to comment it because of the risk of being pedantic, but I see it as something that could make a big difference in the reception and application of your comments.

    • In the background, I'd revise "This STM Working Group was led by…and composed of members of the biggest publishing companies" to be more precise. I don't think the working group members were necessarily from the biggest publishing companies by market share, employee count, revenue, etc. even though several were there, and describing it that way could imply to a less-informed reader that this was an initiative led by big commercial publishers and forced on the community or inadvertently leave out the contributions from smaller organizations that dedicated staff resources to working on the taxonomy. I don't think you mean that (or that it's a reflection of how the taxonomy was developed), so editing the language a little to be more specific would likely help!

    In terms of potential journals for this work, I echo the suggestion of Learned Publishing—it feels to me like its readership is most in line with the audience you're seeking. PLOS ONE and Frontiers in Research Metrics and Analytics may also be welcoming of this kind of comment, though if you're looking for an outlet read by the publisher community, I don't think those would give you the most visibility. If you're not necessarily looking for a peer-reviewed journal to publish in but are rather looking for an outlet where you can maximize visibility, Science Editor could also be worth contacting as well—I know they take pitches! Finally, you may already have done this, but I think it would be worth sharing this with the NISO working group: I know several folks involved in STM working groups (including some folks I see listed in the the group that developed the Taxonomy), and it's a very collaborative, open bunch, so they'd probably be keen to hear your feedback.

    Competing interests

    The author and I volunteer together with an industry organization and have collaborated in the past on a conference session; I was employed by an organization involved in the taxonomy's development and piloting during the time it was being piloted (though I was not personally involved).

  2. PREreview

    This Zenodo record is a permanently preserved version of a Structured PREreview. You can view the complete PREreview at https://prereview.org/reviews/14580016.

    Does the introduction explain the objective of the research presented in the preprint? Yes The initial section after the abstract titled 'Background' clearly and concisely outlines the history of the development of standard terminology for peer review models from the 2019 version 2.0 to the current version 3.0, which is the focus of this commentary. It states the four categories developed in version 3.0 and then states the objective of the commentary which is to explore 3 of the categories, highlight improvements ("gaps") and propose one additional one by recourse to examples of open peer review practice from various disciplines.
    Are the methods well-suited for this research? Highly appropriate This is a commentary piece and as such does not require a methodology section per se but the examples collected to support the author's position are clearly presented in table format, the number and variety of examples are sufficient for a commentary, and the author uses them to great effect to show the nuance and complexity of journal practice, highlighting the need for ongoing revision and improvement of the standards.
    Are the conclusions supported by the data? Highly supported The authors considers 3 aspects of the current standards that could be improved and uses an appropriate number of examples of practice to demonstrate exactly why they are insufficient and exactly how they can be improved - for reviewer transparency by including mandatory and non mandatory visibility by type of reviewer - for reviewer interaction by broadening the category to include non invited public participation - for reviewer information to include the publication of comments. These recommendations are realistic and reflect current practices in open peer review. Five examples of open peer review practice are provided to demonstrate the need to broaden these categories - for a commentary more examples are probably not required but if more examples were sought they would only strengthen the author's position and show even more variety and complexity in journal practice.
    Are the data presentations, including visualizations, well-suited to represent the data? Highly appropriate and clear Using a table format to present the examples employed allows for quick and clear understanding and comparison. Further visualisation is not required for this commentary piece.
    How clearly do the authors discuss, explain, and interpret their findings and potential next steps for the research? Very clearly The need for expansion of categories is made clear from the examples presented, yet the author is also generous in their assessment of the merits of the standards initiative. A couple of the sentences could be proof read as there is awkward phrasing that could result from translation or could be typos such as "a great initiative to support journals and peer review, but not apply to all review models and it is intended" but these are very few and the author writes very clearly and logically and the commentary is very easy to understand. The author concludes with clear realistic recommendations to journals to provide simple explanations of their review process, including visual representations, and to elicit feedback to improve this portion of their websites. In terms of the standards themselves the author suggests constant revision open to public discussion to improve the accuracy of terminology to reflect the evolving practice.
    Is the preprint likely to advance academic knowledge? Highly likely At a time when scholarly publishing practice is innovating and experimenting with new models including peer review, tools to help with consistency and transparency such as the standards are very important and making these tools as effective and accurate as possible is important work. Constantly interrogating such tools to ensure they are accurately reflecting evolving practice and are inclusionary of all forms of new and emerging practice is equally essential. As such this critique is an example of the work required at the coal face to enable the evolution of scholarly publishing
    Would it benefit from language editing? Yes As mentioned earlier there are one or two sentences that require proofreading for grammar to improve clarity but the vast majority of the prose is crystal clear. I would remove the justified format as it does not work in a couple of places with long urls
    Would you recommend this preprint to others? Yes, it's of high quality Yes it is clear, concise, makes well substantiated points and speaks to an important topic.
    Is it ready for attention from an editor, publisher or broader audience? Yes, as it is

    Competing interests

    Possibly. The author is one of a number of guest editors for a special issue for a journal I am editor for. In that regards we have had work related contact with regards to the journal.

  3. Version published to 10.31219/osf.io/h9txu on OSF Preprints