GROUP HAS COMPLETED THEIR WORK AND IS NO LONGER ACTIVE.
FOR QUERIES, PLEASE WRITE TO ‘INFO@FORCE11.ORG’
Crowdreviewing: The Sharing Economy at its Finest
Submitted by: Werner Liebregts
The current review process of scientific articles is outdated. Only a few reviewers assess the quality of papers before publication. Many more experts read them after publication, have a strong opinion about their contents, but are hardly able to share it to a wide audience. Academics would greatly benefit from crowdreviewing, a post-publication peer review (PPPR) process in which anyone who has read a scientific article is asked to review it according to a standardized set of questions. Today’s consumers themselves decide whether the consumed good meets their quality standards, and this is quickly and easily shared with the rest of the world. Others can then let their demand for the reviewed good depend on the reviews. Interested people observe the articles’ quality (and the reasons why) at a glance. Visibility of the reviews, including comments and remarks, will even have an upward effect on the quality of future research. The sharing economy at its finest, applied to academia.
Project Goal
In short: to successfully run a pilot project on crowdreviewing scientific articles.
ScienceOpen has offered me the opportunity to run a pilot project on their already existing platform for open access (OA) publishing, for which I want to thank them a lot.
A major goal of ScienceOpen is to foster the OA movement by raising the awareness for high-quality OA and by involving the entire scientific community in a transparent and open evaluation process. Today, the ScienceOpen platform comprises nearly 1.5 million OA articles and preprints so far, aggregated from various sources.
The associated ScienceOpen network enables authors, readers and reviewers to connect and keep informed about recently published OA articles in their field(s) of interest. ScienceOpen explicitly aims to provide services to the whole scientific community. That is:
- To authors, by enabling them to publish their results immediately, and to receive transparent feedback;
- To readers, by offering them a free to use database of OA articles;
- To reviewers, by giving them the opportunity to receive recognition and credit for their valuable and voluntary duty.
More specifically, I am going to set up a collection of scientific OA articles in economics. A collection should start with a minimum of ten articles. Applying a complete yet accessible review process to it should result in a high number of people reviewing each article in a consistent way. If successful, a new metric for the quality of an individual scientific article might be derived.
Project Outcomes
The outcome of the project is threefold:
- Agreement on how to review scientific articles in a good way;
- A standardized scientific review process suitable for crowdreviewing;
- A first proof of concept.
The first project outcome will be obtained by facilitating and moderating online discussions concerning the scientific review process. The conclusions drawn from the discussions should then be incorporated into the new standard way of reviewing scientific articles. Applying this process to a collection at ScienceOpen will (hopefully) lead to the third project outcome. The success of this project also depends on your cooperation and input!
Project Timeline
Below, you can find a rough timeline of the project, starting with the moment of idea generation in November last year, and ending with a presentation at the next FORCE11 conference.
- November 2014: Idea generation
- November 2014 – end: Idea promotion (incl. personal and Skype meetings with partners)
- February 2015: The £1k Challenge
- February 2015 – end: Contact with ScienceOpen
- March 2015: Editor status at ScienceOpen
- May 2015: Collection set up at ScienceOpen
- May 2015 – August 2015: Online discussions about the scientific review process
- September 2015 – end: Idea implementation
- April 2016: Presentation of the project outcomes (/intermediate results) at the FORCE2016 meeting in Portland
The process of idea implementation comprises the application of an agreed-upon review process to the collection of articles set up at ScienceOpen, inviting readers to review at least one of those articles, monitoring the progress (and making adjustments if necessary), and evaluating the entire process. Short updates about the progress will be published on a regular basis.
Discussions
You are cordially invited to actively participate in online discussions about how to review scientific articles in a good way. Several discussion forums will be opened one by one in the upcoming months. Any comment and/or suggestion is highly appreciated. Examples of questions that will be addressed are:
- What hinders and what encourages people to do a review?
- Can we come up with a standardized set of questions (or, assessment criteria) that are applicable to any type of scientific article from any field of research or should we have multiple ways of reviewing?
- Is it possible to create a one-dimensional indicator of an article’s quality based on reviews?
- What is the minimum number of people to be able to call it a ‘crowd’? In other words, when can the results from crowdreviewing be called reliable?
- Should the reviewing process be fully open? Why (not)?
- Should reviewing be rewarded? If so, how?
