AI/ML Validation Framework Skill

Purpose

The AI/ML Validation Framework Skill supports validation of AI/ML-enabled medical devices per FDA Good Machine Learning Practice (GMLP) principles, addressing data quality, model performance, and predetermined change control.

Capabilities

• Training data quality assessment
• Ground truth labeling validation
• Model performance metrics calculation (AUC, sensitivity, specificity)
• Subgroup performance analysis
• Bias and fairness evaluation
• Predetermined change control plan (PCCP) templates
• Clinical validation study design
• Locked algorithm vs. adaptive documentation
• Model explainability documentation
• Performance monitoring planning
• Real-world performance tracking

Usage Guidelines

When to Use

• Validating AI/ML algorithms
• Assessing training data quality
• Planning clinical validation studies
• Preparing FDA AI/ML submissions

Prerequisites

• Algorithm development complete
• Training/test datasets curated
• Ground truth established
• Intended use clearly defined

Best Practices

• Document data management practices
• Validate on diverse populations
• Plan for performance monitoring
• Consider predetermined change control

Process Integration

This skill integrates with the following processes:

• AI/ML Medical Device Development
• Software Verification and Validation
• Clinical Evaluation Report Development
• Post-Market Surveillance System Implementation

Dependencies

• FDA AI/ML guidance
• GMLP principles
• Fairness toolkits (AIF360, Fairlearn)
• Statistical analysis tools
• Clinical study resources

Configuration

aiml-validation-framework:
  algorithm-types:
    - locked
    - adaptive
    - continuously-learning
  performance-metrics:
    - AUC
    - sensitivity
    - specificity
    - PPV
    - NPV
  subgroup-categories:
    - age
    - sex
    - race
    - disease-severity

Output Artifacts

• Data management documentation
• Algorithm description documents
• Performance reports
• Bias/fairness assessments
• PCCP documents
• Clinical validation protocols
• Monitoring plans
• FDA submission sections

Quality Criteria

• Training data quality documented
• Ground truth methodology validated
• Performance meets clinical requirements
• Subgroup performance acceptable
• Bias assessments completed
• PCCP appropriate for algorithm type