The probability of success of a study is a critical metric in assessing the viability of a study design. In simple trial designs, probability of success can be defined as the study’s statistical power. However, more nuanced definitions of success are available, including some that incorporate assumptions of multiplicity (multiple study arms or multiple endpoints), and uncertainties about the true underlying treatment effect. Assurance is one such Bayesian concept, considering potential variability in treatment effect assumptions.

What is study power?

It is the conditional probability of rejecting your null hypothesis given an assumed treatment effect.

What is assurance?

One way of defining assurance is the expected power across different treatment effect assumptions. Assurance is especially useful when there is uncertainty around the treatment effect. Rather than calculate power based on a single assumption, we can calculate the power across a series of assumptions, and assurance is the average of the power across all scenarios.

In addition, if some information is available about the likelihood of each of these treatment effect scenarios prevailing, it can be incorporated in this calculation to produce a more realistic expectation of results. In this case, each treatment effect scenario is assigned a likelihood, and this likelihood is included in the assurance calculation. This process gives more weight to the scenarios that are more likely to be the true treatment effect.

Cytel has incorporated both concepts — power and assurance — into its East Horizon^TM platform. As well-established metrics in clinical trial design practice, these are widely used in the trial design process. In addition, Cytel has developed two related concepts to power and assurance, performance score and robustness score, to enhance the process of design, offer two additional metrics for assessment, and to elevate the role of the statistician as a strategic thinker in clinical development practice. These two metrics are also embedded in Cytel’s software.

What is the performance score?

The performance score is a linear weighted function that allows the statistician to integrate strategic priorities (reduction in sample size, reduction in duration, and increase in probability of success) into the clinical trial design process. The statistician can assign weights representing the importance of each strategic priority in assessing a study design, and the resulting score provides an estimate of the relative desirability of a design based on these priorities. Thus, the performance score provides an additional metric by which to assess the viability of a study design.

The weighted function of these performance criteria is:

What is the robustness score?

Like the concept of assurance, the robustness score is the average of the performance scores across different treatment effect assumptions. If there is uncertainty about the treatment effect, the robustness score takes that potential variability into account.

Here too, if some prior information about the likelihood of treatment effect scenarios is available, it can be incorporated in the calculation of the robustness score to produce a more nuanced expression of robustness, based on the strategic priorities set by the drug development team. If no prior information is available, each scenario automatically gets assigned equal likelihood.

These four concepts together, power, assurance, performance, and robustness scores, are key tools in the statistician’s toolbox for clinical trial design. Crucially, the latter two metrics are a shift in the statistician’s mindset from a purely scientific consideration of trial design, to incorporating strategic thinking in the design process. In addition, the scoring mechanism allows statisticians to identify a multitude of designs that satisfy basic statistical criteria and choose from among those the best-suited design based on strategic priorities. Finally, the score is a powerful communication tool, anchoring the statistician at the center of the discussion about tradeoffs in priorities in the design selection process with a wider cross-functional team.

In this way, the scoring mechanism embedded in East Horizon has transformed the software from a purely statistical design and analysis tool (albeit nuanced and powerful) to a clinical design strategy tool with a solid and potent statistical core. As more life sciences organizations adopt East Horizon and the advanced design tools it affords, we are seeing a gradual shift in the function of the statistician’s role within these organizations to a more central and consultative role in the trial design process.

Share Via