A Model’s Credibility Is in the Details: FDA Draft Guidance on the Use of AI Models in Drug and Biological Product Development

In January, the U.S. Food and Drug Administration (FDA) issued its first guidance on the use of artificial intelligence (AI)[1] models in drug development and in regulatory submissions titled, “Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products” (Draft Guidance). As FDA noted in its news release announcing the Draft Guidance, the use of AI to produce data or information regarding the safety, effectiveness, or quality of a drug or biological product has increased “exponentially” since 2016, including in drug application submissions over the last several years based, in part, on AI components.[2]

The public comment period is open through April 7.

While, predictably, FDA makes clear that it “does not endorse the use of any specific AI approach or technique,” the Draft Guidance provides a “risk-based” credibility assessment framework intended to establish and evaluate the credibility — or “trust” — of an AI model for a particular “context of use” (COU). It applies to the nonclinical, clinical, postmarketing, and manufacturing phases of the drug development lifecycle. Consistent with FDA’s regulatory authority, it excludes drug discovery and operational efficiencies (think: workflows, resource allocation, the mechanics of drafting regulatory submissions). In other words, the Draft Guidance does not address AI models that do not impact patient safety, drug quality, or the reliability of results from nonclinical or clinical studies.

This article highlights the key takeaways for drug sponsors and manufacturers from this long-awaited regulatory guidance.

1. Adopt FDA’s risk-based framework for assessing AI model credibility.

FDA’s risk-based framework consists of the following seven-step process that it expects sponsors to use to establish and assess AI model credibility:

• Step 1 – Define the question of interest that will be addressed by the AI model. This should describe the specific question, decision, or concern addressed by the AI model. As discussed in the Draft Guidance, an example would be, in a commercial manufacturing context, whether an injectable drug’s vials meet the established fill volume specifications. In the clinical development context, a question of interest might be whether certain clinical trial participants can be considered low risk for a known associated adverse reaction and not need inpatient monitoring after dosing.

• Step 2 – Define the COU for the AI model. The COU provides the scope and role of the AI model used to answer a question of interest. The description of the COU should explain what will be modeled and how model outputs will be used, as well as whether any other information (e.g., animal or clinical studies) will be used in conjunction with the model output to answer the question of interest.

• Step 3 – Assess the AI model risk. Model risk is a combination of model influence (amount of AI model-generated evidence relative to other contributing evidence used to inform the question of interest) and decision consequence (the impact of an adverse outcome resulting from an AI-generated, incorrect output). A greater amount of model influence or decision consequence increases the risk of the AI model and requires more regulatory oversight.

• Step 4 – Develop a plan to establish the credibility of the AI model output within the COU. This “credibility assessment plan” should rely on interactive feedback from FDA about the AI model risk (Step 3) and the COU (Step 2). Early engagement with FDA is recommended to ensure the appropriate credibility assessment activities are adopted based on the particular model risk and COU. Credibility assessment plans should include descriptions of:

• • (A) The Model – Provide AI model inputs, outputs, architecture, features, feature selection process and any loss functions, parameters, and rationale for choosing the specific modeling approach.

• • (B) Model Development Data – Incorporate training data and tuning data. Training data builds AI models by defining model weights, connections, and components. Tuning data explores optimal values of hyperparameters and architectures. Describe the data management practices for the training and tuning datasets and characterize those datasets.

• • (C) Model Training – Explain the AI model’s learning methodology (e.g., supervised, unsupervised), performance metrics and confidence intervals (ROC curve, recall or sensitivity, positive/negative predictive values, true/false positive and true/false negative counts, positive/negative diagnostic likelihood ratios, precision, and/or F1 scores), regularization techniques, and training parameters. Specify whether a pre-trained model was used, describe the use of ensemble methods, explain any calibration, and outline the quality assurance and control procedures.

• • (D) Model Evaluation – Note the model’s data collection strategy, specifying how data independence was achieved and whether there was any overlapping data use between development and testing phases. Include information on the reference method used. Provide information on the applicability of the test data to the COU, the agreement between predicted and observed data (using test data independent of development data), and the rationale for the chosen model evaluation methods. Performance metrics (see model training section above) and limitations of the modeling approaches should also be included.

• Step 5 – Execute the plan. The importance of discussing the plan with FDA before execution to set expectations and to identify potential challenges and how those challenges can be addressed cannot be overstated.

• Step 6 – Document the results of the credibility assessment plan and discuss deviations from the plan. Create a credibility assessment report providing information on the AI model’s credibility for the COU and describing any deviations from the credibility assessment plan outlined in Step 4. The credibility assessment report may be a self-contained document included as part of a regulatory submission or in a meeting package. It may also be held and made available to FDA upon request (e.g., during an inspection). The sponsor should seek FDA’s input regarding whether the credibility assessment report should be proactively submitted to FDA.

• Step 7 – Determine the adequacy of the AI model for the COU. If FDA or the sponsor determines that an AI model is not appropriate for the COU, there are several options for the sponsor moving forward:

• • (A) Reduce the AI model’s influence by adding other types of evidence in response to question of interest.

• • (B) Add development data to increase the model’s output or dial up the rigor of the credibility assessment activities.

• • (C) Create controls to mitigate risk.

• • (D) Update the modeling approach.

• • (E) Classify the AI model’s credibility as inadequate for the COU, which will require model rejection or revision.

2. Prioritize life cycle maintenance — and create a plan to manage it.

The Draft Guidance also highlights the importance of life cycle maintenance, or the management of changes to the AI model to ensure it remains fit for use for its COU throughout the drug product life cycle. Since AI models are data-driven, they can autonomously adapt without any human interventions — and this requires ongoing monitoring. Still, the level of oversight required should correspond to the model risk outlined in Step 3 of the credibility assessment plan.

FDA recommends adopting a risk-based life cycle maintenance plan including model performance metrics, monitoring frequency, and retesting triggers. Quality systems should incorporate these life cycle maintenance plans, and marketing applications should include a summary of any product or process-specific AI models.

Any AI model changes affecting performance should be reported to FDA if required pursuant to applicable regulations.

3. Engage with FDA early and often.

Sponsors and other interested parties should proactively reach out to FDA to clarify regulatory expectations regarding the use of AI models in drug and biologic development. As noted above, early engagement with FDA allows sponsors to set expectations regarding the appropriate credibility assessment activities for the model and identify and address any potential challenges early to ensure they are adequately addressed.

Sponsors may request a formal meeting with FDA to discuss the use of AI in connection with a specific development program. The agency also cites the following engagement options depending on the AI model’s intended use:

• Center for Clinical Trial Innovation (C3TI)
• Complex Innovative Trial Design Meeting Program (CID)
• Drug Development Tools (DDTs)
• Innovative Science and Technology Approaches for New Drugs (ISTAND)
• Digital Health Technologies (DHTs) Program
• Emerging Technology Program (ETP)
• CBER’s Advanced Technologies Team (CATT)
• Model-Informed Drug Development (MIDD) Program
• Real-World Evidence (RWE) Program

Conclusion

FDA’s Draft Guidance provides a helpful roadmap for sponsors and manufacturers navigating agency expectations around AI modeling and drug development.

In summary, FDA has recommended the following steps:

(1) Follow the risk-based framework for AI model credibility;

(2) Create (and follow) a life cycle maintenance plan; and

(3) Engage with FDA about the agency’s emerging regulatory expectations.

On January 23, President Donald Trump signed an executive order “Removing Barriers to American Leadership in Artificial Intelligence” and took steps to rescind the Biden administration’s executive order on AI, which had placed certain restrictions on businesses in an effort to create safeguards for AI development, protecting against issues that may emerge amid automated decision-making in employment contexts, as well as potential worker displacement. This shift, along with changes at FDA based on the new administration, will require careful monitoring of AI policies as they continue to evolve.

If you have questions about the impact of FDA’s Draft Guidance on “Considerations for the Use of AI to Support Regulatory Decision-Making for Drug and Biological Products,” we recommend consulting with legal counsel, including Troutman Pepper Locke.

[1] AI refers to “a machine-based system that can, for a given set of human defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments.”

[2] FDA Proposes Framework to Advance Credibility of AI Models Used for Drug and Biological Product Submissions | FDA; see also Artificial Intelligence for Drug Development | FDA.