BUILDING TRUST IN MACHINE LEARNING: THE NEED FOR TRANSPARENCY IN CLINICAL RESEARCH

Machine learning (ML) innovation in clinical research holds significant promise for enhancing drug development and patient care. However, as with any emerging technology, its growing prevalence often brings increasing skepticism. Pharmaceutical sponsors and clinicians, accustomed to a highly regulated environment that demands rigorous documentation of a drug's development and efficacy, naturally approach ML-powered tools with caution. The concern is valid: these tools often provide little to no insight into how they reach their conclusions—conclusions that may shape the future of clinical trials or impact patient treatment plans. To overcome this distrust, it’s crucial to ensure the transparency needed to help end-users trust ML’s potential to augment their work and believe in its transformative power in healthcare.

In response to the growing call for transparency and standardization in ML, the FDA, in collaboration with international regulatory agencies, introduced Good Machine Learning Practices (GMLP) to standardize the development of safe and ethical ML models. While these guidelines are a promising start towards increasing adoption and trust in these technologies, more is needed. We must go beyond recommendations and establish actionable processes and requirements that provide clear visibility into how these models operate. But before doing so, we must first address some fundamental questions: Why does transparency matter? What exactly do we mean by transparency? And how can developers offer concrete evidence that sound processes were followed in creating a model?

The Need for Transparency to Build Trust

ML’s ability to analyze increasingly complex data points with remarkable accuracy has the potential to revolutionize clinical research and our understanding of healthcare. However, this potential hinges on the assumption that models are developed scientifically and ethically. If not, the validity of their output is not only questionable but may also pose significant safety risks. Much like the “Nutritional Facts” label on food helps us make informed decisions, sponsors and clinicians need insight into the “ingredients” of ML. Without clear oversight into how data is collected, how models are trained and tested, and what drives specific outputs, it’s challenging for end-users to grasp how this technology could impact their patient populations.

Redefining ML with Transparency

To foster transparency, traceability must become a fundamental aspect of ML development. This begins with identifying what aspects of a model should be traceable. Contrary to common belief, the proprietary nature of an algorithm is just a small part of the overall structure. Behind the scenes, there’s an entire system encompassing data selection, collection, testing for relevance and accuracy, and the integration of these elements to produce meaningful outputs. The success of an ML model is more about the robustness of these processes than merely reviewing an algorithm’s code. By gaining insight into how a model functions, sponsors and clinicians might become more willing to integrate ML technologies into their research and practice.

Traceability in Practice

As regulations around ML continue to evolve, developers must anticipate that future requirements may extend beyond the advisory nature of GMLP. They might soon need to prove a model’s effectiveness with certainty, ensuring it was developed safely for its intended patient population. This could involve creating a "pedigree" or audit trail that outlines the workflow of the system and the impact of each component. Furthermore, once deployed, ML technologies must be continuously monitored and refined to ensure they continue meeting the needs of the intended population, with developers held accountable for the model's performance and adherence to the "evidence" provided. Importantly, the average user of this technology may not have a deep understanding of data science, so the information provided must be clear and easily understood.

Advancing Responsible Innovation

Beyond GMLP’s recommendations to improve both the safety and transparency of ML in clinical settings, greater accountability and even incentives are needed to accelerate ML innovation and adoption. Despite their limited collaboration to date, ML developers and regulatory agencies share a common goal: improving patient care and easing the burdens on sponsors and clinicians. By standardizing ML development and enhancing transparency into its operations, we move closer to realizing that goal.