www.BlindingIndex.org

A web-based repository of blinding data from controlled trials

Note: Multiple sets of data tables can be entered for different time points (e.g., shortly after randomization, at the end of trial) and multiple parties (subjects, coordinators, attending physicians). 

Treatment, n

Guess, n
Experimental Control Don’t know
Experimental      
Control      

Aims

To develop web contents for www.blindingindex.org, where clinical trialists used blinding can deposit the data of their studies on the success of blinding.

Background

Why we blind?

Knowledge about treatment allocation and its related expectation and behavior could influence study outcomes in randomized clinical trials. Blinding (or masking) aims to eliminate or minimize this. The importance of blinding has been repeatedly emphasized by regulatory and advisory agencies for clinical trials such as FDA and CONSORT group.1,2 Although blinding, (particularly subject- or therapist- blinding) is not feasible in all disciplines of medicine, studies without blinding are regarded to have inferior quality. While there may be multiple parties to be blinded in trial such as subject, therapist, assessor, and data analyst.

Turning a blind eye to blinding

Despite good effort to implement placebo (sham) control, there is very little evidence that blinding is maintained throughout most trials. In response to this awareness, some clinical trialists reported the results of guess of trial participants regarding the group allocation they were in at the end of treatment or during the early stage of treatments. However, this approach has largely been ignored.3,4 Literature reviews of blinding in clinical trials also called for urgent improvement in this issue.5-8 Blinding-related data were presented in different formats in publication, yet systematic approach to this matter is not commonly adopted, because of the absence of consensus and limited methodological research.

Guessing group allocation at the end of treatment

Sporadically, the data on the "guess" of study participants and those involved in trials revealed that trial participants and staff correctly guessed their group allocation more than can be explained by chance. This indicates that these studies might have failed to achieve blinding. The interpretation of this potential unblinding is an unanswered question.

When participants should be asked for their guesses is another conundrum. Asking before and/or during the early stage of a trial might give important information on participants' perception on the intervention groups, whereas asking at the end of treatment would reflect 'totalistic judgment' of the interventions including their effects.9-12 Theoretically, more frequent information on the participants' guesses may elucidate the detailed dynamics of the participants' judgment and belief during the trial and provide important data on this issue. However, frequent questioning of the participants may trigger their curiosity about their treatment group. This may affect their whole attitude towards the study, influencing factors such as non-compliance and drop-out that could make the treatment effects observed questionable.13 This can jeopardize the internal validity of the trial, and its practicality.

Asking for a guess at the end of treatment will not affect the trial intervention and practicality itself. However, this doesn't guarantee perfect assessment of blinding; the group allocation may have been already unblinded by other factors including therapeutic or adverse effects. By definition, in order for a trial to be blinded, the treatment allocation needs to remain unknown during the entire course of the trial or until the data have been collected.12,14 Objective methods to quantify the guess data and to determine the optimal time for blinding surveys are in great need. In the meantime, it may be sensible to conduct such surveys at the end of study.

Blinding Indexes

To deal with data from these surveys on treatment guessing, James et al.15 and Bang et al.16 developed blinding indexes (BIs). James et al.'s method is a variant of the Kappa statistic, which focuses on disagreement between treatment allocation and guess. Bang et al.'s BI estimates the percentage of correct guessing beyond the level expected by chance in each treatment group. A unique feature in blinding assessment is how to deal with 'Don't know' answer. Since their introduction, both BIs have been used in several studies to quantitatively measure the level of correct guess, a surrogate indicator of unblinding. These two BIs are based on different paradigms and assumptions, and, naturally, they both have their own strengths and weaknesses.17, 20

In our recent editorial,19 we adopted Bang et al.'s Blinding Index (BI) that evaluates the blinding status in each group separately, enabling us to examine different blinding patterns in different arms (i.e. Placebo group and treatment group).17. Bang et al.'s approach allows us to distinguish nine possible scenarios, determined by the combination of three possible results for each of the two arms. The three possible results for each arm are "random" (indicated by values of BI sufficiently close to 0), "unblinded" (indicated by positive BI value, meaning that significant portion of participants guessed their true allocation), and "opposite" (indicated by negative BI values, meaning that significant portion of participants guessed the opposite treatment). From our systematic review of 63 randomized placebo (sham) controlled trials, we classified each trial into 9 (3*3) blinding scenarios.

Most studies unblinded

The most compelling result visible from this table is that about 73.2% (46 out of 63) studies turned out unblinded, mostly in the experimental group19. Each scenario may have different stories, underlying causes and interpretations. While further discussion of the meaning of this finding is a topic of an independent study, the optimistic assumption that blinding remains intact as intended throughout the trial proved to be untrue. This raises several new questions: 1) whether success of blinding should be modeled as a binary variable or as a scale variable; 2) what should be the critical cut off point for BI in order to declare that blinding was successful; 3) which of the above scenarios are desirable to attain satisfactory internal validity; 4) can BI data evaluate the comparability between the groups; 5)what are the impacts of blinding on the study results (including conclusions on the treatment effectiveness/efficacy); and 6) how to understand placebo effect, resurrecting the old question 'Does it have power?' The last question is particularly relevant to the 'wishful thinking' scenario in the table. Answering these questions is more complicated than we may expect, for which much more data and researches are necessary.

Urging to report the BI data

In the middle of more questions than answers, a promising news is that a user-written command for Stata to calculate the two BIs was released in March 2008.18 With this computational tool available, we hope to see more quantitative data and evidence on blinding in the literature, and we hope to see more researches studying the effect of blinding status on the outcomes. We urge all trials claiming "blind" to report the BI at the end of trial and make some efforts to understand qualitative as well as quantitative aspects of (un)blinding and share their stories in research communities, which will shed some light on the internal validity and the quality of trials, providing the communities with new elements to interpret their findings and to plan good clinical trials in the future.

The need for a centralized BI data depository

With anticipated availability of more BI data, it would be sensible to make a centralized web based data depository where more such data can be gathered. BI data may be deposited by its managers as they are identified, or trialists provide the website with either reference information or BI data. This data pool will enable series of studies to look at the association of blinding and other outcomes to be feasible. Providing blinding data is absolutely voluntary.

References

  1. US Food and Drug Administration. Guidance for Industry: Patient-reported outcome measures: Use in medical product development to support labeling claims 2006. http://www.fda.gov/cder/Guidance/5460dft.pdf. Accessed on May 6. 2008.
  2. Rennie D. CONSORT Revised-Improving the reporting randomized trials. JAMA 2001; 285: 2006-2007.
  3. Schulz K F, Grimes D A, Altman D G, Hayes R J. Blinding and exclusions after allocation in randomised controlled trials: Survey of published parallel group trials in obstetrics and gynaecology. BMJ (Clinical Research Ed.) 1996; 312(7033): 742-744.
  4. Fergusson D, Glass K C, Waring D, Shapiro S. Turning a blind eye: The success of blinding reported in a random sample of randomised, placebo controlled trials. BMJ (Clinical Research Ed.) 2004; 328(7437): 432.
  5. Park J, White A R, Stevinson C, Ernst E. Who are we blinding? A systematic review of blinded randomised controlled trials. Perfusion 2001;14: 296-304.
  6. Bebarta V, Luyten D, Heard, K. Emergency medicine animal research: Does use of randomization and blinding affect the results? Academic Emergency Medicine: Official Journal of the Society for Academic Emergency Medicine 2003; 10(6): 684-687.
  7. Boutron I, Estellat C, Ravaud P. A review of blinding in randomized controlled trials found results inconsistent and questionable. Journal of Clinical Epidemiology 2005; 58(12): 1220-1226.
  8. Hrobjartsson A, Forfang E, Haahr M T, Als-Nielsen B, Brorson S. Blinded trials taken to the test: An analysis of randomized clinical trials that report tests for the success of blinding. International Journal of Epidemiology 2007; 36(3): 654-663.
  9. Sackett D L. Turning a blind eye: Why we don't test for blindness at the end of our trials. BMJ (Clinical Research Ed.) 2004; 328(7448): 1136.
  10. Henneicke-von Zepelin H H. Assessment of blinding in clinical trials. Contemporary Clinical Trials 2005; 26(4): 512; author reply 514-5.
  11. Hemila H. Assessment of blinding may be inappropriate after the trial. Contemporary Clinical Trials 2005; 26(4): 512-4, author reply 514-5.
  12. Bang H, Ni L, Davis C E. Response to two Letters to Editor to "Assessment of blinding in clinical trials". Controlled Clinical Trials 2005; 26: 514-515.
  13. Bang H, Davis C E. On estimating treatment effects under non-compliance in randomized clinical trials: Are intent-to-treat or instrumental variables analyses perfect solutions? Statistics in Medicine 2007; 26(5): 954-964.
  14. Colman A M. A Dictionary of Psychology. Oxford: Oxford University Press, 2001.
  15. James K E, Bloch D A, Lee K K, Kraemer HC, Fuller RK. An index for assessing blindness in a multi-centre clinical trial: Disulfiram for alcohol cessation--a VA cooperative study. Statistics in Medicine 1996; 15(13): 1421-1434.
  16. Bang H, Ni L, Davis C E. Assessment of blinding in clinical trials. Controlled Clinical Trials. 2004; 25: 143-156..
  17. Hertzberg V, Chimowitz M, Lynn M, Chester C, Asbury W, Cotsonis G. Use of dose modification schedules is effective for blinding trials of warfarin: Evidence from the WASID study. Clinical Trials (London, England) 2008; 5(1): 23-30.
  18. Chen J. Blinding: Stata module to compute blinding indexes. http://ideas.repec.org/c/boc/bocode/s456898.html. Accessed on May 1, 2008
  19. Park J, Bang H, Cañette I. Blinding in controlled trials, time to do it better. Complementary Therapies in Medicine. 2008. doi:10.1016/j.ctim.2008.05.001
  20. Bang H. Blinding Index for Clinical trials. http://www.med.cornell.edu/public.health/ Slides for Distinguished Speaker Seminar at Univ. of Texas Southwestern, Dallas, TX. April 2008.

This site is being developed by J. Park of University of North Carolina at Chapel Hill in collaboration with Heejung Bang of Weill Medical College of Cornell University, 2008.