Asthma Clinical Trials Versus Real World Effectiveness: Comparing Apples to Oranges

At an industry-sponsored symposium held in conjunction with the 99th International Conference of the American Thoracic Society, three leaders in asthma treatment presented the latest information on identifying types of clinical trial performed and the strengths and weaknesses of the trials. In addition, they explained confounding factors contributing to the differences between clinical trial efficacy and the observed effectiveness of asthma therapies. Finally, they guided participants in interpreting the results of asthma trials and judging their applicability to the care of patients.

This program was supported by an unrestricted educational grant from Merck & Co., Inc.

Clinical Trial Efficacy: What Does It Really Tell You?

Randomized, double blind placebo-controlled trials are the gold standard for determining drug efficacy and safety. Unfortunately, the results of these studies are not always generalizable to broader populations. Understanding the strengths, and at least as importantly the weaknesses, of various kinds of studies is essential to applying information to individual patient therapy.

“Seeing a patient in clinic and considering how to apply what I have learned from clinical trials to the context of individual patient care, I ask myself these questions,” said Craig M. Lilly, MD, Assistant Professor of Medicine, Harvard University in Boston. “Does the endpoint tell me what I need to know in the context of this patient and is it relevant to my patient’s problem? Is the treatment more likely than chance to improve the problem?”

One of the first things to consider is simply looking at what outcomes were investigated. Trials generally focus on things like asthma-free days, exacerbation rates, lung function measures, and asthma-related quality of life issues (Table 1).

“My patients, on the other hand, tend to have combinations of the ‘I-haves’ or the ‘I-can’ts’,” said Dr. Lilly. “The ‘I-can’ts’ are activities that induce symptoms or unwanted sensations. The ‘I-have’s’ are worse.”

Single endpoint trials are popular because they are cheap, easy to perform, and regulatory agencies tend to demand them. Unfortunately, single outcome models have limitations for the practicing clinician.

Simply put, asthma is more than bronchoconstriction.

Clinical trials over the last decade have made it clear; persistent asthma is best controlled with beta-agonists alone. It is also becoming increasingly apparent that some asthmatics have more exacerbations when treated with beta-agonists alone.

Taylor and colleagues randomized patients with mild asthma to receive a placebo, short acting beta-agonist or a long-acting beta-agonist. They were then stratified by beta-adreno receptor status. They found that 25% of those with the ARG-ARG form receptor were uniquely prone to more exacerbations when treated with short-acting beta-agonists (Taylor DR, et al. Thorax 1998;53:744).

“Single endpoint trials don’t tell me all I need to know,” said Dr. Lilly. “Studies reporting efficacy in more than one outcome make it much easier to assess whether the treatment will work for my patients.”

One method to translate clinical trials into optimal asthma care is to discover which trial endpoints are relevant to the patient. He suggests lining up the outcome parameters from various trials and the alternative effective therapies and then fit them together.

“We can now take the treatments and start to look at the patient’s ‘I-have’s’ and ‘I-can’ts’,” said Dr. Lilly.

Placebo effect is an important variable that should be considered when moving clinical trials to clinical practice. Unlike patients, subjects only fail once. This is a primary reason why what happens in clinical trials is different from what is seen in the office or clinic.

To illustrate, Dr. Lilly pointed to a publication by Henry Milgrom and others. Patients were to receive either an anti-IGE therapy or placebo. Symptoms scores after 12 or 20 weeks in treatment were significantly better (Milgrom HR, et al. NEJM. 1999;341:1966).

What was interesting is that the therapeutic effect is relatively small compared to the placebo effect. This indicates that focusing on better environmental control and adherence to therapy actually makes a big difference.

It should also be noted that in these trials symptoms are reduced, but not eliminated. For example, when a leukotriene modifier is added to an inhaled steroid, there was a clinically and statistically significant reduction of 27% in exacerbations. However, there were still patients continuing to have recurrences.

For ethical and humanistic reasons, those enrolled in clinical trials are treated differently when they fail the treatment. They are generally removed from the trial. In this case, the trialist notes that they had only a few failures and the treatment works. In clinical practice, however, those who do not respond to a treatment still require care and often return more frequently with troublesome symptoms.

“They become the gift that keeps on giving, returning multiple times with exacerbations,” said Dr. Lilly. “The clinician sees this as 7 failures in 10 visits and decides that the medication is no good. The challenge is that patients with inadequate responses still need care.”

Clinical Trials: Are These Really Our Patients and Is This Really the World in Which We Practice Medicine?

“Can we really generalize the information from clinical trials to what we are doing every day in the office?” asked Donald Bukstein, MD, Director of Allergy & Asthma Research, Dean Medical Center, Madison, Wisconsin. “If patients in the clinical trial don’t equal the patients in practice, then do the results really reflect what we are likely to see with day-to-day patients?”

There are barriers to physician enrollment in trials that may skew the outcomes. Every time a doctor puts an active patient in a study, there is always a question about how this will affect their relationship. There is also the concern about feeling responsible for new side effects or outcomes if they go into the placebo arm of a study. Loss of autonomy and decision-making authority related to protocols also factor in. Finally, the expense and hassles associated with securing extra informed consents is a consideration (Table 1).

On the patient recruitment side, there are barriers to participation that might change the ability to generalize a study. It takes extra time and cost to participate. Time constraints may impact on planned family outings or vacations.

“These are a few of the barriers that may lead to selection bias,” said Dr. Bukstein.

Gross reviewed 172 trials in high impact journals and found the mean proportion of screened patients to those actually randomized was 65%. Most trials do not identify this figure. If a too small percentage of those screened are randomized and enter into the study, it is prudent to question whether this group actually represents the real world (Gross CP, et al. Ann Intern Med. 2002;137:10).

“You have, in my opinion, research patients and you have clinic patients,” said Dr. Bukstein. “By the time you get randomization, you have eliminated a high enough number of people and have a very select population in the study.”

To address this question, he looked at the patients seen in his clinical practice and two other studies, by Laviolette and Kavuru, that are widely quoted. The percentage of patients with reversibility was 30% in the study as opposed to 14% among Dr. Bukstein’s patients. Compliance was in the high 90% range in the two studies compared to 28% in the clinic group (Laviolette M, AJRCCM. 1999:160:1862; Kavuru M. JACI. 2000;105:1108).

The latter variable is important when measuring real world efficacy. The difference between what was seen in the trial and what is being seen in the practice can impact greatly on how well the average patient does.

That phenomenon can even be seen in the clinical trials themselves sometimes. A study by Jonasson with budesonide continued for 27 months. Adherence was high for the first five months and then tailed off, more indicative of the real world (Jonasson G, et al. Arch Dis Child.2000;83:330).
“Ideal efficacy may be 90% for a medication,” said Dr. Bukstein. “If compliance is 40%, your real efficacy is going to be 36%. A patient might be better off with another medication where the ideal efficacy is less and the compliance is better.”

The chosen endpoints are another part of clinical trials that produce problems. The evaluation boils down to what is the objective of treatment and does the endpoint adequately reflect the treatment being considered? Asthma endpoints have a complex relationship with each other making it difficult to estimate long-term results using one or two surrogate markers.

For example, a study from Israel shows that when comparing the FEV1 results between montelukast versus beclomethasone, there is a better response to the latter. If you look at other endpoints such as asthma attacks, there were no differences. The outcomes you get depend on the endpoints used (Israel E, et al. JACI. 2002;110:847).

“In clinical practice, our endpoints are usually less succinct,” said Dr. Bukstein. “The patient will say they wheeze a couple times a week or they wake up occasionally at night. We can extrapolate clinical study endpoints to practice, but it must be done carefully.”

Publication bias should also be considered. Clinical trials that show positive results are more likely to be published. There are similar indications that industry-sponsored trials are published more often (for example: Bekelman JE, et al. JAMA, 2003;289:454; Dickersin K. Ann NY Acad Sci. 1993;703:135).

How results are reported can further obscure their clinical usefulness. It can be hard from mean values to discover how the results fit into the treatment plan of an individual. In the arginine study, there were three tiers of responders, non-, good, and high. The mean results were much different when they were all put into one pool compared to when they were broken out by response. This may miss subsets of individuals that respond differently from the average and might be helped by a treatment (Greenhalgh T. BMJ. 1997;315:305).

“As the saying goes, I know many statisticians who have drowned crossing a stream with an average depth of six inches,” said Dr. Bukstein. “Clinicians look at individuals while the trials look at groups.”

Statistical significance may not translate into clinical significance. A study may show that a 10-liter a minute change in peak flows is statistically significant. If you look at what that represents on a peak flow meter, the impact is minimal.

There are two main types of study, experimental and observational. Over 50% of the studies seen in major journals are the latter. Most of the literature used to formulate practice guidelines and clinical practice are experimental, randomized trials. Each has their strengths and weaknesses. When looking at trials to steer practice decisions, include many different types with differing designs.

“Clinical trials may not always give you an idea of what is happening in the real world,” said Dr. Bukstein. “Yet randomized trials are the mainstay of medical advancement. Observational studies and other types of information are important to mix in as you develop your belief systems and how you are going to treat individual patients.”

Considering Therapeutic Choices in The Real World: How Do We Apply This To Our Practice?

There is a difference between clinical trials and clinical research that doctors should remember when evaluating the literature and how it fits in their practice. Most pharmaceutical clinical trials are designed to get drugs approved by regulatory authorities. Clinical research looks for answers to specific questions.

“Our guidelines approach has seductively led us to believe that asthma can be simplified,” said Stanley Szefler, MD, Head of Pediatric Clinical Pharmacology, National Jewish Medical and Research Center, Denver. “In practical terms, most of our concerns center on just two, symptoms and airflow limitation.”

Even in this rather truncated universe, there is still great variation at the individual treatment level. There are many things, such as allergen levels, activity levels, and symptom recognition, influencing the day-to-day course of the patient.

When a patient comes into the office, the focus is on finding out just how bad their disease is. Therapy is then geared toward correcting that.
One aspect is pulmonary function. Usually treatment consists of a course of steroids that is eventually tapered to a satisfactory maintenance level. Some do not respond even to high dose regimens. Using information from patients in his studies, Dr. Szefler wanted to know if they could find differences between the responders and those who did not.

Roughly one-third of those in a recent study receiving either beclomethasone or fluticasone metered dose inhalers demonstrated no differences in FEV1 despite changes in dose and method of delivery. Those that had a greater percent improvement in FEV1 had higher exhaled nitric oxide, greater bronchodilator reversibility and a low FEV1 /FVC ratio. The maximal improvement in pulmonary function after about 8 puffs of albuterol increased to only 84% in the non-responders (Szefler SJ, et al. J Allergy Clin Immunol 2002;109:410).

The FEV1 response was near maximal with a medium dose of medication. By pushing the dose you did not get more airway changes, but there were systemic side effects from the steroids.

For measures of change in airway hyper-responsiveness a similar pattern was observed. This time poor responders tended to have a longer duration of disease and childhood onset along with higher sputum eosinophils and exhaled nitric oxide.

“From this data physicians should be able to make some guesses about how likely you are to see an improvement in these measures of asthma control,” said Dr. Szefler.

Disease progression prevention is another aspect of asthma where important subgroups may be hiding in the mean responses of clinical trials. Progression can be defined as the need to escalate supplementary therapy over time.

“As we followed patients in a long-term study who had good control, about 25% had a loss of function over time, irrespective of the treatment arm,” said Dr. Szefler. “Clearly our medication algorithms do not have drugs identified that alter decline in function.”

Researchers were given access to clinical trial database sets from the Pivotal Trials for fluticasone. When looking at severity and improvements in pulmonary function, there was a good improvement in the most severe asthmatics. However, this improvement still topped out at around 65%. Even in the mild asthmatics, there was room for improvement (Szefler SJ, et al. J Allergy Clin Immunol. 1999;103:780).

There are a large number of new medications either released in the last few years or subject for approval within the next few years. Integrating them into clinical practice will hopefully advance asthma care.

A recent study used methacholine responsiveness to help make decisions about selecting and adjusting asthma therapy. Another study utilized sputum eosinohils. Both studies showed a reduction in asthma exacerbations by applying these tools to asthma management (Sont JK, et al. Am J Respir Crit Care Med. 1999;159:1043; Green RH, et al. Lancet. 2002;360:1715).

Genetics is another area of intense interest. However, there will probably be no single gene that predisposes an individual to asthma or determines response to therapy.

“How are we going to put this whole package together for the patient?” asked Dr. Szefler. “We will be challenged to look at asthma characteristics, biomarkers, and genetics to identify the profile of asthma severity, and to select the medications most likely to benefit the patient.”