Apples and oranges? Interpreting success in HIV prevention trials

Published: August 7, 2010


In the last decade, several large-scale, clinical trials evaluating the efficacy of novel HIV prevention products have been completed, and eight are currently underway or about to be reported. Little attention has been given in the literature to the level of protection sufficient to warrant introduction, and there is concern that using the term “efficacy” to describe the effect of user-controlled methods such as microbicides may mislead policymakers.

We review how the fields of family planning, vaccine science and mathematical modelling understand and use the terms efficacy and effectiveness, and explore with simple mathematical models how trial results of user-controlled products relate to common understandings of these terms.

Each field brings different assumptions, a different evidence base and different expectations to interpretations of efficacy and effectiveness — a reality that could cloud informed assessment of emerging data.

When making judgments on the utility of new health technologies, it is important to use standards that yield appropriate comparisons for the innovation and that take into account the local epidemic and available alternatives.

Keywords: Efficacy; Effectiveness; HIV prevention; Vaccine; Microbicide; Contraception; Family planning
Article Outline


Efficacy vs. effectiveness
    2.1. The meaning of efficacy and effectiveness in trial results
    2.2. Efficacy in the world of mathematical modelling

Current evidence of the efficacy of different HIV and contraceptive technologies
    3.1. Efficacy of HIV prevention methods
    3.2. Efficacy of contraceptive methods

Explaining partial protection

What do trial results tell us about product efficacy?

Interpreting trials: how good is good enough?

Appendix A.


1. Introduction
Over the last decade, several large-scale, clinical trials evaluating the efficacy of various novel HIV prevention products, including experimental vaccines, use of oral HIV drugs for pre-exposure prophylaxis (PrEP) and a variety of new microbicide candidates for preventing HIV transmission to women during vaginal sex, have been reported, and seven more are underway or about to be reported [1]. The trial designs include a wide range of effect sizes for the sample size calculation (33–60% reduction), but it is not clear what level of protection will be deemed sufficient to warrant introduction. This will depend in part on how policymakers interpret the numbers emerging from clinical trials and how these compare to their expectations regarding efficacy and effectiveness. Given that condoms are routinely cited as 95% efficacious, for example, some HIV stakeholders perceive anything less as unacceptable.

In assessing “how good is good enough”, it is important to differentiate between user-controlled and provider-controlled methods, particularly in relation to the impact on estimates of biological efficacy that can be derived using trial data. There is also confusion in translating the results from trials into individual risks which are often easier for people to understand when deciding whether or not a product will work for them.

As a first step toward understanding these issues, we review the definitions and use of the terms efficacy and effectiveness across different fields, including vaccine science, HIV prevention, family planning and mathematical modelling. Then we use mathematical modelling to illustrate how these various measures relate in the particular case of HIV transmission and attempt to determine the appropriate standards for use with user-controlled methods.
2. Efficacy vs. effectiveness

With respect to health outcomes, efficacy is the improvement achieved with use of an intervention by participants under ideal conditions. It frequently refers to research settings or situations of perfect use. By contrast, effectiveness is reserved for the effect that can be achieved in practice, taking into account limited coverage, constrained resources and inconsistent or imperfect use.

Both efficacy and effectiveness are measures of relative rather than of absolute risk; they compare the incidence of a health outcome among individuals receiving an intervention vs. those who do not and are expressed as ratios. The absolute risk by contrast is expressed as a simple proportion.
2.1. The meaning of efficacy and effectiveness in trial results

For interventions like vaccines, where adherence by trial participants can be measured objectively, it is often possible to get an accurate estimate of biological efficacy from the results of a Phase 3 trial, provided loss to follow-up is minimal (see Appendix A for how vaccine efficacy is calculated). This is generally based on a per-protocol analysis, restricted to those who complete the full immunization schedule (perfect use).

When evaluating products that are user dependent, however, the trial results cannot be adjusted to derive efficacy unless there are reliable biological measures of product adherence. In trials of products such as microbicidal gels or PrEP, not all participants will use the product correctly or consistently, and measurement of use relies on self-reports. A Phase 3 trial therefore provides a combined measure of the product’s biological efficacy and the pattern of use in the trial (adherence). Indeed, because of incorrect and inconsistent use, a trial that demonstrates a 40% reduction in incidence of HIV between its arms necessarily implies a product of higher efficacy.
2.2. Efficacy in the world of mathematical modelling

Modellers use the term “efficacy” differently to denote risk reduction for a single act of sexual intercourse with an infected partner rather than the protection achieved over time (as in trials). They use this per-sex-act efficacy in combination with many other factors — including the consistency with which it is used in different partnerships; the probability that the woman’s sexual partner is HIV infected; whether he is in the high viremia phase; and whether either partner has other sexually transmitted infections — to project how use of a method may affect patterns of HIV transmission.

For outcomes that are relatively rare, like HIV, per-sex-act efficacy is a reasonable approximation of the efficacy of the method as understood by trialists or program planners. The same is not true, however, for more infectious pathogens, such as bacterial STIs, which have a higher transmission probability [2]. Thus, it is not always appropriate to equate the parameters used in modelling articles with the estimates of efficacy derived from clinical trials, although for HIV the values are comparable.
3. Current evidence of the efficacy of different HIV and contraceptive technologies

When the trial results for any new HIV prevention method become available, donors and policymakers will have to decide whether it merits further investment and ultimately introduction.1 Their evaluation of the potential utility of any new method will depend in part on their assessment of the adequacy of existing options, their reading of available evidence and the assumptions they may bring from past experience.

Current evidence on the efficacy and effectiveness of different prevention technologies varies widely. Likewise, the fields of HIV prevention, family planning/reproductive health and vaccine science hold different working assumptions about what level of efficacy warrants introduction and use.
3.1. Efficacy of HIV prevention methods

In the world of HIV, the strongest evidence for an effective prevention strategy exists for male circumcision, which has been shown in three randomized controlled trials to reduce the risk of HIV acquisition among heterosexual men by roughly 50% to 60% [3], [4] and [5]. The evidence for other HIV interventions is considerably less rigorous. For example, there have been no randomized controlled trials to assess the efficacy of the male and female condom against HIV and other STIs, although there have been several reviews of the available data [6] and [7]. Available estimates come from observational cohort studies of HIV discordant couples where a sero-negative person with a known exposure can be followed over time. When some individuals use condoms 100% of the time and some never use condoms, efficacy is calculated by taking one minus the ratio of the HIV incidence among “always users” of condoms vs. “never users”, based on self-reports.

A Cochrane meta-analysis of these studies suggests a reduction in risk with consistent condom use of 80%, with the confidence limits for “consistent users” ranging between 34% and 94% [8]. When inconsistent users are also considered, condoms reduce the annual HIV incidence by 69% [9].

Reviews of condom effectiveness have also sought to estimate the per-sex-act efficacy of the condom. In particular, Pinkerton and Abramson [10] used a Bernoulli model to “back calculate” the likely per-sex-act efficacy of male condoms associated with observed reductions in incidence among discordant couples. They concluded that condoms reduced the probability of HIV transmission per sex act by as much as 95%. It is this per-sex-act modelled figure that is often cited for both condom efficacy and effectiveness (see Table 1).
Table 1.

Estimates of condom efficacy and effectiveness

Full text of article available at link below –

Leave a Reply