Discussion

DISCUSSION

The evolving attitude towards HT, from a treatment many believed unethical not to prescribe to post-menopausal women to one associated with negligible benefit and moderate risk, is a good example of how scientific thinking about a drug may change over time. The HT debate illustrates the various scientific meth-ods used to understand drug safety and how each can either increase understanding or produce confusion. While it is difficult to make decisions on HT use now because of uncertainty, the evolution of knowledge about HT provides a good example of current thinking about the tradeoffs of using data from observational studies or clinical trials to understand the population impact of drug treatment. Some of these principles are discussed below using examples from the study of HT.

STRENGTHS OF CLINICAL TRIALS  COMPARED TO OBSERVATIONAL EPIDEMIOLOGY STUDIES

Despite their strengths, which make them valuable tools in attempting to determine relationships between drugs and outcomes, observational epidemiologic studies have a number of limitations. These limita-tions are a large part of the reason that clinical trials remain the gold standard for evaluating drug-outcome relationships.

A major concern with observational studies is that they are not randomized and treatment may well vary according to characteristics linked to the disease process. This could occur through selective prescribing or through issues related to medical care access. Confounding by demographics, socio-economic status, and variables related to health, as described previously, can be a substantial problem in observational studies.

A second problem with observational epidemiology studies, especially the larger ones, is that they tend to have less robust ways of evaluating endpoints than clinical trials, relying on self-report or simple clin-ical reports, rather than a methodical AE reporting system. Short-term effects of treatment may be partic-ularly difficult to ‘capture’ in observational settings. There may also be inconsistent endpoint definitions between studies, making comparability difficult. This problem may have contributed to the divergent esti-mates of CHD in women taking HT in the NHS and Framingham studies, which defined CHD differently.

In addition to problems ascertaining outcomes, observational studies also frequently have poor or limited characterization of health status or lifestyle factors, particularly those that change over time. For example, over-the-counter medications that can affect outcomes are not always ascertained and variables such as physical activity levels and diet are often not measured as often or with as much precision as would be desirable. Because of potential differences in these factors between those prescribed and those not prescribed drugs, residual confounding is likely to be present in observational studies. If some of the rele-vant confounding variables are measured at baseline (and, if possible, throughout the study), adjustment for some confounding is certainly possible, but residual confounding remains likely.

Another potential limitation of observational studies is assessment of drug exposure, in terms of both dose and duration. The duration of use of a drug is some-times poorly defined in observational studies. Often treatment use is measured at the beginning of the study or only at irregular periods throughout the study, and constant use is assumed, whether this is valid or not. Observational studies often have poor information on the dose or particular formulation of a drug that is being used. Strategies, including having patients bring all of their medicines with them to intake visits, have been developed to help with this problem, but many studies, particularly those that are survey-based, have limitations related to exposure assessment. Clinical trials, on the other hand, frequently employ systems such as pill counts or blood level monitoring that allow researchers to monitor actual dose received on an ongoing basis.

Finally, in observational studies, particularly cross-sectional, case-control and prevalence studies, it is often not possible to establish a temporal relation-ship between drug and disease, which is crucial to establishing cause-and-effect relationships, rather than simple associations. This limitation can impede inter-pretation, such as when a drug improves survival with a condition or increases its latency period, both bene-fits, rather than causing the condition itself.

STRENGTHS OF OBSERVATIONAL EPIDEMIOLOGY STUDIES COMPARED TO CLINICAL TRIALS

As shown by some of the analyses from the WHI, confounding can be a problem even in clinical trials, especially when the blind is imperfect. This can lead to some problems of differential follow-up and ascertain-ment, but these are usually less prominent than those seen in observational epidemiologic studies. Critics cite this as a potential flaw of the WHI that may, in addition to other factors (such as the age structure of the population the trial was conducted in), have biased the results.

Though the WHI looked at hard outcomes, many clinical trials use surrogates as their primary outcomes of interest. This can cast doubt about whether the results of such trials are clinically meaningful. By contrast, observational epidemiologic studies can, and usually do, look at ‘real’ events (such as MI) rather than their surrogates (e.g., cholesterol). This is due in part to the fact that, unlike clinical trials, they can be retrospective (mitigating the need for costly follow-up) or long-term prospective follow-up of a large-scale cohort may be feasible.

The WHI, which looked at actual events in a large population of women over a long time period, had many of the advantages of a large-scale prospective follow-up usually associated with observational stud-ies rather than with clinical trials, but duration of follow-up tended to be shorter than in many observa-tional studies. However, some of the earlier, smaller clinical trials that form an important part of the evidence base about the risks and benefits of HT used surrogate endpoints and were limited to very short-term follow-up. Thus, the duration of obser-vational studies is frequently longer than is feasible for clinical trials, which allows evaluation of ‘hard’ outcomes instead of surrogates and of rare or time-delayed effects. Part of the controversy following the results of the early, short-term randomized controlled trials of HT stemmed from their short duration (espe-cially compared to some of the observational studies) and, in some trials, use of surrogate markers rather than clinical disease states for some outcomes.

Another strength is that observational studies occur in ‘real life’, using drugs in the particular dose and schedule used by patients in the field. This may more appropriately represent usage patterns than more controlled studies. One criticism of the WHI is that only one combination of oestrogen and progestin was evaluated, though other doses and formulations exist. Arguably, the effects of a drug, in terms of both risks and benefits, may vary by the dose, duration of expo-sure and route of administration, as well as the demo-graphics and health status of those treated. Thus, an observational study may be better equipped to eval-uate several factors that are relevant to usage of the drug in ‘real life’.

Finally, because of their decreased demands in terms of cost, observational studies, particularly large simple studies, often allow for larger sample sizes and longer duration of follow-up than clinical trials. Although some consider observational studies more ‘cost-effective’, tradeoffs related to confounding and selection biases must be given careful consideration, as discussed above.