AI reliability: An interactive decision tree.

Last updated February 15, 2024

Proper data collection and curation strategies are crucial in medical image analysis within a data commons, such as MIDRC, are critically important to yield reliable AI algorithms. Issues can arise in many steps along the AI/machine learning (ML) pipeline, from data collection to model deployment in clinical practice. Our AI reliability tool brought to you by the AI Reliability Working Group was created to aid in identifying potential problems and it provides descriptions, impacts, real-world examples, measurement methods, mitigation considerations, and literature references for the following steps along the medical image analysis AI/ML pipeline:

  • Data Collection

  • Data Preparation and Annotation

  • Model Development

  • Model Evaluation

  • Model Deployment

While our AI reliability tool may not provide an entirely comprehensive list, the interactive tool below includes about 30 sources of potential issues found in the 5 main steps along the AI/ML pipeline (including many that can impact multiple phases).

Please cite: Drukker, K., Chen, W., Gichoya, J., Gruszauskas, N., Kalpathy-Cramer, J., Koyejo, S., Myers, K., Sá, R.C., Sahiner, B., Whitney, H., Zhang, Z., and Giger ML, 2023. Journal of Medical Imaging, 10(6), pp.061104-061104. https://doi.org/10.1117/1.JMI.10.6.061104.