How to review a journal paper

At some point during your PhD or in your years after your PhD, you will be asked to review a paper. I’ve done a fair amount of reviews by now, and have started to keep track of the reviews I write about a year ago by using Publons. You can see my Publons profile here.

If you receive an invitation to review paper, you need to ask yourself the following questions:
1. Do I have sufficient technical knowledge to review this paper? If not, can I recommend a colleague?
2. Do I have enough time to write this review by the deadline?
3. Do I have a conflict of interest that prevents me from writing an unbiased report?
If you have the time and knowledge it takes to review the paper, and no conflict of interest, you can go ahead and review the paper.

As you read the paper, you need to keep in mind how you will review the paper. A typical review report follows a certain standard form. If you know which elements you need to discuss in your review report, you can pay attention to these while you read the paper. Besides your standard written review report, you may also be asked to fill out an evaluation form on the review website. This post deals only with the basic elements your review report should contain.

A review report usually contains the following elements, often in this order as well:

1. The general information. You can write for example on the first line “Comments on a paper submitted to Journal“, followed by the title and then the Manuscript ID.
2. The first heading should be “General comments”. In this section you write your review report, in which you focus on the strengths and weaknesses of the presented study.
3. The first paragraph should be a summary in your own words of the manuscript. You can also discuss the possible readership for the manuscript. Is it interesting for researchers, or can it be of value to practitioners?
4. In the next paragraph, discuss if the introduction introduces the topic in the appropriate manner. If not, give suggestions for improvement.
5. Then, discuss the literature review in a paragraph. Are all topics discussed in the manuscript adequately placed within the literature with a thorough literature review?
6. Discuss the methods. Which methods are used? Are the methods appropriate? Are the methods explained in a clear manner? Can you spot weaknesses in the applied methods?
7. How did the authors interpret their results? Are they providing a satisfactory explanation for their observations? How do these observations fit within the body of knowledge of your research field. Are the results used for the development of recommendations? Are these recommendations practice-ready. If there are gaps in the interpretation or possibilities for implementation, point these out.
8. Evaluate the summary and conclusions section. There should be no new information in these sections, and they should be clear for reading.
9. Discuss the writing/language. If the manuscript suffers from serious editorial issues, suggest the authors to send it to a professional proofreading office.
10. Discuss the figures and tables. Are they clear? Do they follow the guidelines of the journal?
11. Write a conclusion of your review report. Summarize in one paragraph your decision (accept, minor revisions, major revisions, reject) and give the main reasons for this decision.
12. The second heading of your review report should read “Technical/Editorial comments.” Add a table with detailed technical and editorial comments below your general comments section. You can use the following columns: Page – Lie – T/E – Comment to organize your more detailed comments.

If you want some more inspiration about how to review a paper, you can read the guidelines of Hugh Davis, Shriram Krishnamurthi , and Barak Pearlmutter. Veronika Cheplygina focused on how to become a reviewer, and Science magazine has an interesting article with the experiences of different scholars in reviewing journal papers.