Health technology assessment (HTA) bodies support decision-making for the reimbursement of new technologies at the local or national level. Recommendations made by HTA bodies are based on various sources of evidence, ranging from the preferred standard randomized clinical trials to real-world data (RWD) when trials are unavailable or not relevant to the target population of the decision problem. Non-randomized studies of treatment effects are already widely used in rare diseases and innovative technologies to contextualize findings from single-arm trials. Watch our recent webinar on real-world external control arms here.

To build trust in the evidence that supports decision making, researchers need to understand and address potential risks to study validity.

Study validity: Internal vs external validity of treatment effects

Study validity refers to the accuracy and trustworthiness of a study’s results in answering a specific research question for a target population of interest. The concept applies to all types of studies, however, when estimating treatment effects, study validity is demonstrated through two concepts:

Internal validity: This refers to the unbiased estimation of the causal effect within the study population from which inferences are drawn. It is a reasonable assumption for randomized trials, but it needs to be justified in the case of non-randomized studies that use observational data.

External validity: This refers to how well the causal effect generalizes to a different setting (i.e., the target population of interest).

Most of the causal inference literature focuses on strategies to address internal validity bias in observational studies, arising from the presence of potential unmeasured confounding. Although internal validity is of utmost importance, whether evidence is generalizable to a target population becomes very relevant from an HTA perspective as decisions need to be made for specific target (local) populations. RWD can potentially increase external validity compared to clinical trials. However, what happens when international RWD is used to make decisions in local settings?

External validity: Transportability vs Generalizability

When we define external validity, we talk about how our study findings generalize to a target population. But how does generalizability differ from transportability? The difference between generalizability and transportability stems from how we define the target population of interest. In the first case, the study population (e.g., participants in a clinical trial or observational study) could be a subset of the target population (all patients that would be eligible for a specific intervention). In the case of transportability, the target population is entirely different (non-overlapping), for example, patients in a different country that were not participating in the study. In both cases, we are addressing a generalizability problem, but “transportability” or “transferability” is becoming a widely-used term.

What do HTAs think: Do local data matter?

With the increasing use of data from different countries, decision makers globally have issued guidance that makes explicit mention of considerations for the transportability of non-local data to their jurisdictions. In the US, for example, the AHRQ[1] and FDA[2] guidance documents, some of which date back over a decade, suggest justifying the selection of non-US data and understanding the similarities and differences between US and non-US systems. Similarly, other countries consistently emphasize the need for a clear justification when using non-local observational data. It is interesting to see that NICE[3] and CADTH[4] made a specific mention that non-local data can be an acceptable source depending on the context, while other agencies such as the German IQWIG lean toward a more conservative position.

Concerns about transportability of evidence from non-local studies stem from patient or setting differences that may have an impact on estimates that are important for reimbursement decisions. Relative effects (e.g., hazard ratios) are typically not at risk of external bias compared to absolute estimates (e.g., overall survival estimates for a given treatment).

What can we do to address transportability concerns?

While a good understanding of transportability elements is important, are there methods available to identify and correct for potential external validity bias? Transportability analysis methods are quantitative approaches, similar to those used to adjust for confounding bias, that help to reliably extend conclusions from one study population to an external target population. It can involve extending conclusions from trial populations to real-world target populations (which is the focus of most current literature) or between non-overlapping real-world populations. Of note, NICE recently updated their RWE framework with external validity considerations and an overview of endorsed methods to minimize external validity bias.[3]

Although transportability methods have been often used to extend conclusions from clinical trials, there are very few demonstration projects applying these methods between observational studies. Cytel previously performed a transportability study of overall survival from US to Canadian lung cancer populations.[5] Broadly, there are key elements that one needs to define in a transportability analysis as we illustrate below.

High-level steps for conducting a transportability study. For further reading on the theory of transportability and methods, we suggest reading the overview by Degtiar and Rose. [6]

Final Takeaway

With increasing use of non-local RWD in HTA submissions, it’s important not only to address concerns about internal study validity but also to consider the transportability of evidence to the target population. In addition to differences in patient-level characteristics between the two compared populations, probable differences in health care systems (care delivery, patient adherence, or staff experience) and their settings can be difficult to adjust for in transportability analysis. Use of statistical models to quantify the impact of differences in patient or setting characteristics on study estimates, can provide support for the transportability of real-world evidence (RWE ).

References

[1] Agency for Healthcare Research and Quality, Developing a Protocol for Observational Comparative Effectiveness Research: A User’s Guide. 2013.

[2] Food and Drug Administration, Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products Draft Guidance for Industry. 2021.

[3] https://www.nice.org.uk/corporate/ecd9/resources/nice-realworld-evidence-framework-pdf-1124020816837

[4] https://www.cadth.ca/guidance-reporting-real-world-evidence

[5] Ramagopalan SV, Popat S, Gupta A, Boyne DJ, Lockhart A, Hsu G, O’Sullivan DE, Inskip J, Ray J, Cheung WY, Griesinger F, Subbiah V. Transportability of Overall Survival Estimates From US to Canadian Patients With Advanced Non-Small Cell Lung Cancer With Implications for Regulatory and Health Technology Assessment. JAMA Netw Open. 2022 Nov 1;5(11):e2239874. doi: 10.1001/jamanetworkopen.2022.39874. PMID: 36326765; PMCID: PMC9634498.

[6] Degtiar, I., & Rose, S. (2023). A review of generalizability and transportability. Annual Review of Statistics and Its Application, 10, 501-524. https://doi.org/10.1146/annurev-statistics-042522-103837

Join Cytel at the 2024 ISPE Conference, where we will be showcasing our latest advancements in clinical trial design, biostatistics, and real-world evidence. Click here for the complete ISPE program.

Book a meeting with us to discover how our expertise can drive your projects forward.

Share Via