What is this study all about?
A working group within Cochrane called the Rapid Review Methods Group (RRMG) is leading this work to assess how reliable abstract screening is when only a single person does it (I told you it was exciting). They and us, on the Cochrane Crowd team, have designed a study that will hopefully help to answer that question. The study is made up of three ‘arms’ – three different groups that participants to the study could be assigned to. If you already have some experience of screening search results for a review, then you will be randomly assigned to screen a set of records retrieved from a search for either an RCT-based review (a review that is including only RCTs) or for a public health review that has broader study design inclusion criteria. If you have no experience screening for a specific review, you will be assigned to the machine+crowd arm where you can screen as many records as you like for the RCT-based review.
What do you mean by the machine+crowd arm?
As well as testing single vs dual screening (one person vs two people screening independently), we also want to look at the performance of Cochrane’s machine learning RCT classifier (‘machine’) and the accuracy of the crowd approach which uses the crowd agreement algorithm (lots of people screening independently) to help reach accurate collective decisions.
What exactly is the task I’ll get assigned to?
If you get assigned to the RCT-based review, your task will be to find RCTs that look potentially relevant to the review question. If you get assigned to the public health arm, your task is to find studies (not just RCTs in this case) that look potentially relevant to that review. Each arm is supported by a mandatory training module. The training takes you through everything you need to know and will take around 15-20 minutes to complete. For arms 1 and 2 you will then have 100 records to screen. For arm 3 (machine+crowd arm), there is no limit so screen as many as you like!
What support or guidance is there for me to do the task I’m assigned to well?
Apart from the training module, which you can re-do as often as you like, there is also a task-specific FAQ sheet, and a Quick Reference Guide. Both can be accessed from the task dashboard once you have been assigned to one of the study arms.
This all sounds great, how do I join this study?
Hopefully very easily! Just click on this http://crowd.cochrane.org/singlescreeningstudy.php. This will launch the participant questionnaire. Once completed, you will be assigned to one of the three arms. Simply log-in to Cochrane Crowd and you should see the task near the top of your dashboard. It will look something like this:
You will then need to complete the training module. You need to get 12 or more correct out 15 to progress to the live records. However, don’t panic, you can re-take the training as often as you like.
What if I make a mistake?
You are bound to make the odd mistake. Even experienced screeners misclassify the odd record. That’s why we’re hoping to power this study adequately (i.e. have enough people participate) to give us a clear indication of the reliability of single-screening versus dual-screening versus crowd-screening.
How long will this task be around for?
This task will be live until Friday 14th June. We may extend it if we feel we need to.
How long will this task take?
The two main arms of this study contain 100 abstracts to screen. We estimate this will take around 2 to 3 hours to finish. However, you don’t have to do it all in one go. The deadline for the project is 14th June so if you’ve only signed up a week before, then you’ll need to try and do around 20 records per day. A perfect task to accompany a nice cup of tea. Also, if you don’t quite manage to get all 100 done by the end of the study, it’s not the end of the world. We will still be able to use your contribution.
What if I sign up and then realise I don’t really want to do it after all?
If you sign up and then find you don’t want to do this after all, or you simply don’t think you have the time, then that is absolutely fine. You don’t need to feel bad or to notify us. We expect that some will sign up and then realise that actually it’s not for them after all.
What do I get out of this?
You can win one of five Amazon Vouchers (worth 100 US$ each). Additionally, if you agreed to it in the survey, we will acknowledge your contribution by name on a scientific poster at the next Cochrane Colloquium if it is accepted.
Will personal data be collected?
Yes, but only data that you have provided in the questionnaire. By answering the questions in the questionnaire and screening records in Cochrane Crowd you are giving your permission for us to use any data we collect in accordance with the Cochrane general terms and conditions available here. If you have changed your mind and you don´t want to get your data collected, that is totally fine. Unfortunately, you can't participate then. Let us know at firstname.lastname@example.org.
Can I feedback my thoughts to the project team?
If you have any questions, queries or suggestions in the meantime please don’t hesitate to email either Anna or Emily at email@example.com. If you put Methods study in the subject line, that would be helpful (but don’t worry of you forget).
Happy citation screening and thank you very much for giving this a go!