monitor the safety and effectiveness of medical devices once they are on the market. For SaMD, this involves rigorous data collection and analysis post-launch to preemptively identify safety signals and trends that may not have been evident during pre-market evaluations.

According to the FDA’s Guidance on Post-Market Surveillance Under Section 522 of the Federal Food, Drug, and Cosmetic Act, manufacturers are expected to establish comprehensive PMS systems to detect issues and mitigate risk effectively. As digital health technologies rapidly evolve, particularly with the integration of AI, it is imperative for stakeholders to create a robust framework to monitor and adjust their products based on real-world evidence and user feedback.

Core KPIs for Evaluating SaMD Post-Market Safety Systems

When establishing KPIs for SaMD PMS, it is essential to select metrics that are both specific and measurable. The following categories of KPIs can help assess the effectiveness of post-market safety systems:

1. Complaints Handling

One of the most crucial aspects of PMS is efficient complaints handling. The effectiveness of a complaints management system can be evaluated using the following KPIs:

• Response Time: Measure the average time taken to acknowledge and respond to user complaints. A shorter response time indicates a more nimble and effective complaint handling process.
• Investigation Resolution Time: Track the time taken to investigate and resolve complaints. This can provide insights into the adequacy of initial complaint assessments and subsequent actions taken.
• Complaint Closure Rate: Evaluate the percentage of complaints that are resolved satisfactorily within a specified timeframe. A high closure rate is indicative of an effective complaints handling process.

By meticulously tracking these KPIs, organizations can optimize their responses to user feedback, thus enhancing the overall safety and user experience of their SaMD products.

2. Safety Signals Detection

Safety signals refer to emerging indications that suggest a possible relationship between the SaMD and adverse events. The following KPIs are vital in assessing how well a SaMD safety system can detect these signals:

• Signal Detection Time: Measure the average time taken from the identification of a potential safety signal to its reporting and investigation. Quick detection is critical for timely response and risk mitigation.
• Number of Signals Confirmed: Track the total number of safety signals that are formally confirmed through further investigation. This will highlight the effectiveness of the monitoring system in identifying genuine issues.
• Signal Analysis Findings: Document the findings from safety signal analysis, focusing on trends and patterns, to inform strategic decisions on software updates or modifications.

Timely detection and analysis of safety signals allows SaMD developers to promptly act on potential risks, aiding compliance with regulatory expectations.

3. Software Recalls and Field Corrections

Recalls and field corrections provide insights into the effectiveness of a SaMD’s quality assurance protocols. Evaluate the following KPIs:

• Recall Rate: The total number of software recalls in relation to the total number of units sold. This metric signals potential safety or performance issues that need immediate attention.
• Field Correction Effectiveness: Measure the success rate of field corrections implemented to address product issues. This will help assess the adequacy and efficiency of corrective actions.
• User Feedback Post-Correction: Track user feedback after implementing corrective measures to evaluate their continued confidence in the SaMD product.

Monitoring recalls and field corrections provides critical insights into the product lifecycle and helps manufacturers maintain compliance with both FDA and international regulatory standards.

4. AI Model Changes

The increasing use of AI in SaMD necessitates distinct KPIs to monitor the effectiveness of AI model changes on product performance:

• Accuracy of AI Predictions: Assess the accuracy of AI predictions pre-and post-model adjustment. This KPI will help determine whether changes improve the device’s clinical effectiveness.
• User Engagement Levels: Analyze the changes in user engagement metrics following AI model updates. Increased engagement may suggest improved user experience and satisfaction.
• Adverse Events Linked to AI Changes: Evaluate whether there is any correlation between AI model changes and reported adverse events. Understanding this relationship helps improve future AI updates.

Effectively managing AI model changes is essential, given that even minor adjustments can significantly affect the device’s performance and user safety.

Implementing an Effective KPI Monitoring Program

Integrating KPIs into a SaMD post-market safety system requires a systematic approach:

1. Establish Clear Objectives

Before implementing KPIs, define the specific objectives of the post-market safety system. Are you looking to improve user safety, enhance product performance, or reduce recall rates? Clear objectives will guide the selection of relevant KPIs.

2. Data Collection and Management

Establish a reliable data collection framework. Consider leveraging software tools for data collection, analysis, and reporting to ensure data accuracy and accessibility. Data privacy and security must comply with regulations, especially given the sensitivity around health data.

3. Regular Monitoring and Review

Provide regular oversight of the collected data. Set up periodic reviews to assess KPI performance and identify trends. This ongoing process enables organizations to adapt to safety signals and user feedback more effectively.

4. Stakeholder Engagement

Involve stakeholders across various departments, including clinical, quality assurance, regulatory, and technical teams, in the KPI monitoring process. Collaboration across these areas will provide comprehensive insights and foster a culture of safety and quality.

5. Continuous Improvement

Utilize KPI findings to inform adjustments to SaMD products and processes. Cultivating a mindset of continuous quality improvement supports a dynamic post-market safety environment that can effectively respond to emerging risks.

Regulatory Expectations for SaMD Post-Market Surveillance

For stakeholders involved in the development of SaMD, understanding regulatory expectations is crucial. In the US, FDA regulations under 21 CFR 806 emphasize the necessity of timely reporting regarding recalls, safety issues, and adverse events. Moreover, the FDA’s Digital Health Innovation Action Plan explicitly outlines the importance of PMS in the digital health realm, which aligns with similar initiatives in the EU and UK.

The EU Medical Device Regulation (MDR) and the In Vitro Diagnostic Regulation (IVDR) enforce strict post-market obligations as well. Organizations must ensure compliance with these regulations by systematically monitoring product performance and safety, as non-compliance carries significant consequences.

Conclusion

The evaluation of KPIs related to post-market surveillance, field actions, and software updates for SaMD is indispensable for maintaining product safety and effectiveness. By implementing structured frameworks that encompass complaints handling, safety signals detection, recalls, and AI model changes, digital health organizations can safeguard user health while ensuring regulatory compliance.

Aligning these processes with US FDA expectations while also recognizing international standards sets a foundation for sustainable success in the evolving landscape of digital health and medical software. Engaging a collaborative, continuous improvement mindset is essential for responding to the dynamic nature of medical software and ensuring optimal safety for users.