Frequently asked questions for RCT identification

What tasks are available on Cochrane Crowd and what’s in it for me?

We have the RCT identification task, and a diagnostic test accuracy test accuracy identification task. We are also piloting two RCT identification tasks that are focused on screening search results retrieved for a specific review. We are offering these two pilot tasks to anyone in Crowd who has screened 200 or more citations in the RCT ID task. For these tasks we are exploring Crowd performance, in terms of accuracy and efficiency and the potential role of rewards and incentives. If you screen 250 or more records you will be offered acknowledgement in the review. The top screener will be offered authorship. Authorship is not a small reward and does bring with it some responsibility. The potential author will also have to fulfil Cochrane’s authorship and conflict of interest policies http://community.cochrane.org/editorial-and-publishing-policy-resource/ethical-considerations/conflicts-interest-and-cochrane-reviews and http://community.cochrane.org/editorial-and-publishing-policy-resource/ethical-considerations/authorship-and-contributorship#criteria.

 

The decision will not be based solely on the quantity you screen for this exercise. Accuracy will play a critical role.

Why is the RCT identification task important?

We need these citations screened. They feed a unique and very important product to the world of evidence based medicine: the Cochrane Central Register of Controlled Trials, CENTRAL. This product helps us to find trials quickly and easily which in turn helps to answer vitally important questions about the effects of a treatment. The more up-to-date and comprehensive CENTRAL is, the more effective we can be in answering those questions quickly and accurately. Considered as a task on its own, it can be difficult to see just how important it is, but it is fundamental to what we do. Without identifying the trials, we cannot appraise them, assess them, meta-analyse them; we cannot accurately answer those questions about the effectiveness of new treatments or interventions.

 

To find out more about CENTRAL and how it is created go to: http://www.cochranelibrary.com/about/central-landing-page.html

 

To find out more about national access to the Cochrane Library go to: http://www.cochranelibrary.com/help/access-options-for-cochrane-library.html

Where do the records come from that the community work on?

In the future, records will come from a range of sources, some freely accessible such as from ClinicalTrials.gov or PubMed. However some of our sources, now and in the future, will be proprietary databases. At the moment we are working on records that have been sourced from Embase (via the Ovid SP platform). Embase is owned by Elsevier, a for-profit publisher with whom our publisher, Wiley has a contract. This enables us to access and re-publish those records in CENTRAL. Wherever possible we will seek to share the records identified by the community with as wide an audience as possible, within a framework that acknowledges and respects the legal rights and efforts of database providers.

What if I make a mistake?

You are bound to make the odd mistake, especially when you first start doing this. Even experienced screeners misclassify the odd record. We have created a very robust agreement algorithm to help counter this so don’t worry. No single citation is screened by only one person and disagreements are arbitrated by expert screeners.

Can I go back and change a classification?

Yes, you can go back and change a classification if you think you have made a mistake. However, this will only be doable if the record is still active – that is, before the record has received enough classifications for it to not require any more.

What exactly should I be looking for?

For this task, we’re looking for reports of randomised controlled trials (often shortened to RCT) and sometimes also referred to simply as randomised trials, or controlled trials that may have been randomised or used a quasi-randomisation technique, such as allocating participants to treatment or placebo based on date of birth (which we refer to as CCT).

A randomised controlled trial is a particular way to run an experiment. The randomised controlled trials that we are interested in are those that are looking to test the effects of an intervention (the intervention might be a drug, a device, a form of therapy etc). The experiment goes like this: a number of people with a similar condition, for example 100 people with Type 2 diabetes, are recruited into a study. Each person within the study is randomly allocated to the intervention (a new drug or a new device or a new therapy) or a comparator (a sugar pill/placebo, or quite often just treatment as usual or another treatment). If the trial is also ‘double-blind’ then the 100 people in the trial don’t know whether they have the new drug or the comparator, and the people giving out the drug don’t know either. The random allocation and the blinding help make this kind of experiment as fair as possible.

If you are interested in finding out more about what a randomised trial is and why it is considered the gold standard way to conduct a clinical trial, you might like to view this 5 minute video which describes well what a randomised controlled trial is: http://www.ecranproject.eu/en/content/sail-along-james-lind

Can I re-do the training?

Yes, you can. Simply click on the Training button in your task dashboard to access the training records.

Can I see what the final decision was for a record?

Yes, if you look at My history you’ll be able to sort by the records where the final decision was not the same as the classification you gave. This can be a really useful way to learn. You can also see all the records where your classification was in agreement with the final classification – and you can feel very smug about that!

Can I choose to screen particular citations?

Yes, you can do this. You need to go to your Settings. There you will see a setting called: Prioritise the records I receive. Enter a word or short phrase that describes what you are interested in e.g. dementia, and we will prioritise records containing those terms for you. If you don’t get records containing any of those terms, it is because we don’t have in the current batch being screened.

What if I sign up and then realise I don’t really want to do it after all?

This will rip a hole in the space-time continuum and it’ll be hard to put right. Of course, I’m only joking. If you sign up and then find you don’t want to do this after all, or you simply don’t think you have the time, then that is absolutely fine. You don’t need to feel bad or to notify us. We expect that some will sign up and then realise that actually it’s not for them after all.

How many records do I need to screen each day?

You can screen as many as you like. We are not in any danger of running out of records to screen. It would be great if you got into doing around ten minutes per day. That would really make a significant difference to this project.

There are words/phrases highlighted Yellow or Red – what are these?

We have around 80 words or phrases highlighted red or yellow. The yellow highlights are considered likely to appear in records we are looking for (i.e. records reporting randomised controlled trials). Therefore records with lots of yellow on them are likely to be describing an RCT or CCT. However, don’t instantly classify it as RCT/CCT – the yellow words and phrases are only helping to make sensible suggestions – they don’t guarantee that the record is an RCT/CCT. The red highlights are considered likely to appear in records that we are not looking for – the Reject records. However, again, don’t instantly classify a record as a reject just because it contains a red highlight.

Can I add my own highlights?

Yes, you can add your own highlights. You do this in the Settings section. You can add as many as you like, and you can colour code your highlights as well. They are not visible to other screeners.

What is the Notes feature on the screening page?

The Notes feature allows you to add notes about a record. These notes are visible to you and to Resolver screeners. The task of Resolver screeners is to reach a final decision on a record where there have been conflicting classifications or where there has been an Unsure classification.

Sometimes when I am screening, I only see a title and no abstract. What should I do?

Some records don’t have abstracts (summaries), they just have a title. If you cannot tell from the title whether it is describing a randomised trial, then click Unsure and select Not enough information to decide. If you think it is likely that it is describing a randomised trial, then click RCT/CCT, and if it seems very unlikely then click Reject.

Can I feedback my thoughts to the project team?

We, at Project Transform, would love to hear your feedback on any aspects of Cochrane Crowd. Please don’t hesitate to contact me: Anna Noel-Storr either at anna.noel-storr@rdm.ox.ac.uk or crowd@cochrane.org

Can I screen on my tablet or my smart phone?

Yes, the screening tool is web-based but works well on tablets and on phones.

Can I work offline?

Yes. The way Cochrane Crowd works is that a small number of records are downloaded for you automatically. These are then automatically synced with the server. If you know you’ll be offline for a while and want to do this task, then you can select to download up to a 1000 records to work on. This feature is accessible from your Settings.

I’m a trainer/educator, can I use this task in my lessons?

We are developing a tool that will enable trainers and educators to use this task and others in their classroom or learning environments. If you are interested in piloting this new tool for us, we’d like to hear from you, so please do get in touch by emailing me: anna.noel-storr@rdm.ox.ac.uk

What do I get out this?

Screening citations may sound like a very simple and dull task. However, it is a really excellent way to become more familiar with the current biomedical literature and the research that is going on. I have screened tens of thousands of citations since joining Cochrane in 2008 and I am certain it has helped me increase my knowledge and understanding of evidence based medicine in my area of work (dementia). You will also be helping Cochrane in a task that needs doing. Cochrane is an international not-for-profit organisation. Our work is internationally recognised as the benchmark for high quality information about the effectiveness of health care. By helping us, you will become a part of that effort to produce high-quality, relevant, accessible systematic reviews and other synthesised research evidence.

Where can I find out more about the results of this work?

We have done a number of evaluations of our methods and processes looking at things like accuracy and engagement. Here is a link to an oral presentation that aims to give a fairly detailed overview of our findings to date 
(https://www.researchgate.net/project/The-Embase-project_572c53f33d7f4b078367fef1), or email us at crowd@cochrane.org and we will be happy to send you a copy.

Are there any other similar tasks I could help with?

New tasks are coming soon! Watch this space.