Session:  New models for evaluation for research and researchers

Session organizers:  Carole Goble

When:  Wed, March 20th 13:00-14:30

Description:  Online research and publishing gives us opportunities to revise how we evaluate research and how we evaluate researchers.  For the evaluation of research, reproducibility underpins the scientific method. For an experimental finding to be reproducible, its materials must be available and its methods clear, accurate and transparent. Increasingly within the scholarly community (academics, funders, publishers, industry) there is a demand that published results are not just reported as reproducible but can be evaluated as such by independent third parties. This demand is being addressed by a range of models: from reproducibility certification (e.g. https://www.scienceexchange.com/reproducibility) through to computational platforms for re-running codes (e.g. http://runmycode.org) and executable papers.  For the evaluation of researchers, the cost entailed by supporting reproducibility must be balanced by rewards. New (alt)metrics are needed to discriminate between reproducible and merely reported research in order to impact reputation and credit, and drive cultural change. There must be rewards, too, for the provisioning of reproducibility platforms and for peer-evaluating in addition to peer-reviewing.

The session aims to produce a roadmap of the steps needed to realise these new models of evaluation. The session is structured around a role-playing panel who will discuss several scenarios. Each panelist represents (several) communities, will be sceptical as well as positive, and will stay in their roles. The audience will participate by role-playing and scenario proposing too. There will be no formal presentations.

The panelists are:
– Carole Goble (Manchester) : chair, technical engineer, realist, tool provider
– Scott Edmunds (GigaScience): publisher (old school and new school)
– Christine Borgman (UCLA): digital librarian, scholarship gatekeeper
– Jan Reichelt (Mendeley): scholarly comms vendor, commercial funder
– Victoria Stodden (Columbia): policy maker, government, public funder
– Phil Bourne (PLoS, UCSD): institutional dean, promotions chair, hard pressed editor in chief and reviewer, data service manager
– Steve Pettifer (UTOPIA, Manchester): young academic, enthusiast

Presentation: Click Here to download the PPT