Part 2: Protocol Design

Stakeholder Sample Considerations

The protocol design should describe the methodology for surveying key stakeholders and analyzing the results. Designing a protocol that addresses stakeholder recruitment in a manner that achieves a representative sample, while minimizing selection bias, is one of the greatest challenges facing a REMS assessment. A REMS assessment should be a reliable prediction of actual market behavior based upon prescription events. It should predict performance of the REMS at the level of filled prescriptions. Health care providers (HCPs) vary greatly by prescription volume of the drug in question. Thus, for assessing a health care professional communication plan, the HCP sample should weight preference toward higher volume prescribers since they make up a larger portion of the prescription universe. In fielding to patients, it is also important to weight preference to patients prescribed the drug from a higher volume prescriber since these prescribers have proportionately greater public health exposure by virtue of their number of drug treated patients. Specific know how for prescriber sampling techniques requires both experience and expertise in working with a range of third party data sources.

In summary, the ideal representative sampling will have decile 10 prescribers and their patients comprising about 10% of the survey sample, decile 9 prescribers and their patients comprising another 10% of the sample, and so on. Following this method, deciles 3 to 10 would comprise about 80% of the sample. In order to achieve this sampling mix, special techniques must be utilized to ensure sample recruitment reflects this decile weighting over a series of repeated measures. Since there are fewer prescribers in the top deciles and a greater number of prescribers in the lower deciles, on a percentage basis per decile, a larger share of top decile prescribers will be represented in the survey.

For a prescriber stakeholder survey, we suggest restricting your prescriber and patient sample to only those prescribers who are defined in the REMS as a target for a HCP communication plan in order to measure effectiveness of the program.

Recruiting Stakeholders

The methodology for recruiting key stakeholders should result in achieving a very similar sample for each assessment reporting period since this is a repeated measures study design. This requires thoughtful planning during protocol development. Many pharmaceutical companies are exasperated by the thought of recruiting patients for a REMS assessment survey, and for good reason. Reaching patients in a manner that achieves a representative sample of the prescription events for a particular drug is a difficult and expensive undertaking, but nonetheless, necessary. In my experience, engaging representative prescribers is a starting point for recruiting patients. For unique patient populations, it may be necessary to add other methodologies such as utilizing support from patient advocacy groups, registries, and organizations. Print and radio advertisement is also an option, but can be costly and yield limited results.

Sample Size and Statistical Considerations

I highly recommend powering the study according to the performance thresholds defined for the individual measurement domains of interest. The statistical plan, expected measurement range, and desired sensitivity will determine the sample size needed for reliable analysis. Our preferred statistical method for measuring comprehension and understanding of product risks and communications distribution compliance is a binomial test: Pass/Fail or Yes/No based on test score and correct/incorrect for individual questions or incidence. We have found that a reportable sample size of 150 stakeholders is usually adequate for each REMS assessment reporting period based on this statistical plan. For example, you might establish a REMS performance goal of achieving Patient Understanding of Risks and Appropriate Actions at 80%. Meaning, at least 80% of patients had a passing score based upon a defined domain of key risk assessment questions. This approach is applicable for any stakeholder group and domain of interest and meets FDA’s draft guidance for REMS assessments.

Survey Content

In designing the survey instrument itself, first determine the various methods by which the survey instrument will be fielded – electronic or web-based, paper survey, or both. It is safest to limit content to only asking those questions that relate to the objective of the assessment – that is, to specifically measure the domain areas of interest. For example, patient survey items 1, 2, and 3 may be comprehension and understanding of key drug risk questions that as a composite provide a measurement for that domain. Item 4 might be a separate compliance measure. We design REMS assessment surveys to measure these or other specified domains of interest as an approach to predicting the current effectiveness of the REMS elements.

Here are some helpful hints when designing the REMS survey instrument:

• Avoid use of questions that can be leading, educating or obvious in response intent.
• Electronic survey formats can reduce the potential for error and open up a lot of possibilities, such as displaying images of Med Guides or communication materials.
• Establish domain performance thresholds and design questions to drive the domain analytics.
• It is highly recommended to perform pretesting and validation of the survey instrument.

What about surveys for non-English speakers? In our experience, respondents whose first language is something other than English typically score lower on surveys of a technical or scientific nature, including patient assessments. This important issue should be given attention as there is a large and growing multi-ethnic population the United States.

Survey Fielding and Timing

Now that a range of REMS assessment topics have been covered including survey design, I would like to discuss fielding the REMS survey. Ideally, fielding of the REMS survey should not occur at one time point only, but rather reflect observations throughout the reporting period. For example, if your REMS has an 18 month reporting interval, it is a best practice to collect data during at least 10-12 months of this period in order to reflect the effectiveness of the REMS program during the overall reporting period. Of course, you will first want to launch your REMS program elements and tools before measuring their effectiveness and this can delay survey fielding.

Pharmaceutical manufacturers required to design and implement a REMS for a particular drug are best served to use a third party for survey fielding and data collection. This keeps an arms-length distance from data collection and ensures that personally identifiable information received from patients and health care professionals remain protected.

In summary, it is easy to see why the pharmaceutical industry has struggled with methodologies for conducting a REMS assessment. I hope this article has been helpful and provided a deeper level of understanding about REMS assessment. Please do not hesitate to contact me if I can be of further help to you.

Larry A. Risen
President
BioTrak Market Intelligence, Inc.
Carlsbad, CA

About the Author: Larry Risen has over twenty-four years of health care product experience in commercial planning and research. Mr. Risen is founder and president of BioTrak Market Intelligence, Inc.. Since 1999, BioTrak has fielded hundreds of survey projects to patients and prescribers and has established industry best practices in REMS assessment design and implementation. You can reach Mr. Risen at risen@biotrak.com or visit his Company website at www.biotrak.com if you wish to communicate with him about your REMS questions or consulting needs.