(SaMD): This document outlines the FDA’s expectations regarding the safety and effectiveness of software-based products, including AI systems.

ICH Q8-Q11 Guidelines: These guidelines address pharmaceutical development, quality risk management, and the control strategy pertinent to using AI and ML in product development.

EMA Reflection Paper on Big Data in Medicines Development: The EMA’s guidance offers strategic insights into how big data, including AI, can be optimally leveraged within the regulatory framework.

Legal and Regulatory Basis

In the US, the regulatory framework is governed primarily by the Food, Drug, and Cosmetic Act (FDCA) and its associated regulations, particularly 21 CFR Parts 803 (Medical Device Reporting), 820 (Quality System Regulation), and 11 (Electronic Records; Electronic Signatures). In the EU, the Medical Device Regulation (MDR) and In Vitro Diagnostic Device Regulation (IVDR) establish stringent requirements for software classification and performance evaluation. The UK, following its exit from the EU, has implemented similar regulations under its Medicines and Medical Devices Act.

Compliance with these regulations necessitates that organizations conduct thorough vendor qualification audits to ensure that AI and ML quality platforms adhere to the principles of data integrity, risk management, and transparency.

Documentation Requirements

Effective documentation is critical for passing regulatory audits and inspections. The following documents should be prepared and maintained as part of the vendor qualification process:

• Vendor Quality Agreement: This should outline the responsibilities, expectations, and quality metrics for the vendor.
• Vendor Qualification Protocol: A detailed approach to assessing the vendor’s compliance with regulatory requirements, including the methodologies used for audits.
• Scorecard or Checklist: A structured document that reviews key areas such as GxP compliance, data security measures, and disaster recovery plans.
• Audit Reports: These should capture findings from the qualification audits, including any identified deficiencies and corrective actions taken.

Review and Approval Flow

The review and approval process for AI vendor qualification generally follows a systematic flow. This can be summarized in the following steps:

1. Identify Vendor: Based on needs assessment and technology requirements.
2. Conduct Pre-audit Evaluation: Review vendor credentials, prior audits, and market reputation.
3. Perform Qualification Audit: Comprehensive on-site or remote assessment based on established checklists and protocols.
4. Evaluation of Audit Findings: Determine compliance status and document any areas for improvement.
5. Final Assessment and Approval: Formally approve the vendor based on the outcome of the qualification audit and agree on ongoing oversight mechanism.

Common Deficiencies in Vendor Qualification Audits

In reviewing common deficiencies noted during vendor qualification audits for AI and ML systems, regulatory professionals must be aware of the following areas:

• Insufficient Algorithm Transparency: Vendors need to provide clear documentation of their algorithms, detailing how the AI/ML models are trained and validated.
• Lack of Data Integrity Measures: Ensuring the integrity of data generated by AI tools requires rigorous control mechanisms and process validations.
• Inadequate Risk Management Practices: Failure to establish robust risk assessment frameworks tailored to AI solutions can lead to compliance issues.
• Poor Vendor Oversight: Continuous monitoring and re-evaluation processes are essential to maintaining compliance and ensuring ongoing efficacy.

Decision Points in Vendor Qualification

Regulatory Affairs professionals must navigate several decision points during vendor qualification audits. The most critical include:

When to File as a Variation vs. a New Application

Understanding when to submit a variation to an existing application versus a new application is crucial for compliance:

• Variation: Generally appropriate when changes to an existing vendor’s AI system do not significantly alter the intended use, safety, or effectiveness. Document any proposed alterations and rationalize their impact through a submission to the regulatory agency.
• New Application: Required if the changes involve new technology that significantly impacts the product’s risk profile or intended use; a new vendor’s AI/ML solution often necessitates a new application.

How to Justify Bridging Data

A common requirement during the review of vendor qualifications involves justifying the use of bridging data:

• Demonstrate Similarity: Provide data showcasing how the bridging data relates closely to the original source while assessing equivalence in performance characteristics.
• Conduct Comprehensive Comparability Studies: Undertake studies supporting that the outcomes derived from the new vendor’s AI system are comparable with established benchmarks or historical data.

Practical Tips for Documentation and Responses to Agency Queries

Agencies expect detailed and precise responses to queries, especially in the context of AI vendor audits. The following tips can help streamline documentation and submission processes:

• Keep Documentation Updated: Regularly review and update all vendor documentation, reflecting any changes in technologies, regulations, or organizational policies.
• Proactively Address Common Queries: Prepare for typical questions related to algorithm efficacy, data integrity, and vendor oversight by maintaining thorough documentation and justifications ready for agency review.
• Ensure Multidisciplinary Collaboration: Foster communication between regulatory affairs, Quality Assurance (QA), Clinical teams, and Commercial functions to provide comprehensive evidence and responses to agencies.

Conclusion

Vendor qualification audits for AI and ML quality platforms present a unique set of challenges and opportunities for regulatory professionals. By adhering to the regulations set forth by key authorities such as the FDA, EMA, and MHRA and following best practices for documentation and vendor oversight, organizations can effectively navigate the complexities of this evolving landscape.

For further reading on the regulatory guidelines regarding software as a medical device and AI technologies, refer to the FDA’s official guidelines. Continuous adaptation of practices in alignment with regulatory expectations will not only ensure compliance but also enhance the quality and reliability of AI-driven solutions in the life sciences.