How we found 15 million hours of lost time

Lost time in the current peer review process

Rubriq wants to recover lost hours from redundant reviews so they can be put back into research. In the current journal submission process, rejection is common, yet reviews are rarely shared from one journal to the next.  Even when reviews are passed along, the lack of an industry-wide standard means that each journal in the chain solicits its own reviews before making a decision. All of this leads to reviewers repeating work that has already been done on the same manuscript by other colleagues.  We estimate that over 15 million hours are spent on redundant or unnecessary reviews – every year. 

Here’s a video that helps illustrate the key issues:

(once it starts, you can click on “HD” in the right-hand corner to view on highest resolution)

So how did we get to that number of 15 million hours each year?

The two key metrics for finding wasted time are quantity (how many manuscripts are reviewed and then rejected?) and time (how long does each submission one take to be reviewed?). While there are 28,000 peer reviewed journals, we only use 12,000 in our calculations since that is roughly the number of high-quality journals that are included in Thomson Reuters’ Web of Science.  The figure below shows how we calculated both quantity and time, and the descriptions and citations for the key steps in the process follow:

Rubriq calculation of time lost to peer review - click to view larger

Rubriq calculation of time lost to peer review – click to view larger

 

Calculation & Source Details:

1.  3,360,207 (English-language, STM) submissions per year

  • Although the MarkWare STM report1 showed that there are over 28,000 peer-reviewed journals, we focused our scope within just the 12,000 English-language STM journals identified in that same report, as they are the current focus for Rubriq.
  • The average submissions per journal were shown in the Thompson Reuters data2 as 280 (total ScholarOne submissions divided by the count of ScholarOne journal sites). Calculating 280 submissions for each of the 12,000 journals equals 3,360,207 submissions per year.
  • Note that this is submission-based data, not paper-based.  A single manuscript that was rejected by one journal but then accepted by another within the same year would go through two review cycles and thus recognized as two separate submissions.

2.  1,344,099 (40%) accepted submissions per year

  • Thompson Reuters data2 reports 37% acceptance based on all submissions received and accepted within their system, but the MarkWare PRC report3 estimated an average of 50%.
  • We feel the Thomson Reuters data is more accurate than PRC data based on how the information was collected and how calculations were made. Combined with our own internal data and personal interviews with some of the largest STM publishers, we selected 40% as the best representation for this group of journals. 40% of our total submission number equals 1,344,099 accepted papers.

3.  705,652 (21%) submissions per year rejected WITHOUT Review

  • The MarkWare PRC report3 stated 21% as its estimate for submissions that are rejected without going through peer review, also known as a “desk rejection”.
  • Although there is time lost and an opportunity cost to the author when this occurs and they have to try again with another journal, we are currently only focused on time spent on peer-review, so do not factor this group in with our calculation of wasted time.

4.  1,310,496 (39%) submissions per year rejected WITH Review

  • The number of submissions that are sent to peer review but are then rejected is our key starting metric for calculating lost hours (why? See our “Additional Reading” section below for some background material). We use the two preceding calculations to find this number.
  • If 21% were rejected without review, and 40% were accepted, then the remaining submissions were rejected after the peer review process.  Applying 39% to our total gives us 1,310,496.

5.  11.5 average reviewer hours spent per submission

  • Data from the MarkWare STM report4 provided us with an average (median) of five hours spent per review.
  • The MarkWare PRC report3 states that an average of 2.3 reviewers is used for each submission.
  • Five hours * 2.3 reviewers equals 11.5 average review hours per submission.
  • Note that this number only takes into account the time spent per submission by reviewers – it does not include time spent by the journal or publisher in coordinating the review process (e.g., recruiting reviewers, editorial check of reviews, review software costs) or other time spent processing these papers (e.g., screening, editorial review, technical check, other operational time).

6. 15,070,706 hours per year spent on redundant reviews

  • Assuming 11.5 hours per submission * 1,310,496 submissions that were reviewed but then rejected = over 15 million hours. Every year.
  • Since there are only 8,760 hours in a year, you can also think of it as equaling 1,720 years (if it was all one reviewer working 24 hrs per day).

 

 References/Links:

1. M. Ware, M. Mabe, The STM Report: An overview of scientific and scholarly journal publishing (International Association of Scientific, Technical, and Medical Publishers, Oxford, UK, 2012; http://www.stm-assoc.org/2012_12_11_STM_Report_2012.pdf)

2. Thomson Reuters, Global Publishing: Changes in submission trends and the impact on scholarly publishers (April 2012: http://scholarone.com/about/industry_insights/).

3. M. Ware, Peer review: benefits, perceptions, and alternatives (Publishing Research Consortium, London UK, 2008; http://www.publishingresearch.net/documents/PRCsummary4Warefinal.pdf)

4. Mark Ware (2011): Peer Review: Recent Experience and Future

Directions, New Review of Information Networking, 16:1, 23-53 http://dx.doi.org/10.1080/13614576.2011.566812

 

Have other questions? Found a better number with your own calculations? Feel free to add your comments here on our blog!

 

2 Comments

Filed under Uncategorized

2 responses to “How we found 15 million hours of lost time

  1. Pingback: The Peer Renew System | I, Science

  2. Pingback: PEER REVIEW WATCH

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s