The EMBASE project provides an opportunity for new and potential contributors to get involved with Cochrane work by diving into a task that needs doing. No prior experience is necessary as the task supports a ‘learn as you do’ approach.
The project's purpose is to identify reports of randomised controlled trials (RCTs) and quasi-RCTs from EMBASE for publication in the Cochrane Central Register of Controlled Trials (CENTRAL). It is run by a team from Metaxis Ltd, (developer of the Cochrane Register of Studies), the Cochrane Dementia and Cognitive Improvement Group, and York Health Economics Consortium (YHEC).
A crucial part of the project was to develop and implement a screening task, and the innovative bit is that this task is crowd-sourced. A web-based screening tool has been developed so that anyone, with access to the internet, can join the collective effort to screen the search results for relevance within CENTRAL. A quality-control system has been developed so that all records will be viewed by at least two screeners. Records viewed by ‘novice’ screeners will need three consecutive agreements on the record’s relevance for it to then be either published in CENTRAL or ‘rejected’. Disagreements will be arbitrated by experts. All new screeners have to complete a small, interactive test set of records before progressing to ‘live’ records.
This task has been designed so that it fits around people’s busy lives and they can dip in and out as suits them, very much like the classification tasks offered by the likes of Zooniverse and other citizen science initiatives. We don’t want volunteers to experience the burden of working to a deadline or to feel that they cannot adjust their level of involvement.
As by the end of April, a total of 232 volunteers have signed up and screened more than 38,000 records identifying 1147 RCTs or quasi-RCTS. Early data on the accuracy of the crowd and the robustness of the algorithm is extremely positive with 99.8% sensitivity and 99.8% specificity.