Being able to evaluate a machine learning (ML) model is essential part of the toolbox for hospital administrators, providers, and insurance administrators. In this paper, we first discuss ML algorithms, of which there are several important components to consider when evaluating them, including bias testing and metric selection. We then focus on metric selection and some commonly used tools to evaluate the performance of a supervised binary (two classes) classification model, illustrating them through several example scenarios. Our discussion includes: 

• ML model performance evaluation 
• Applications of ML model evaluation metrics
• Additional considerations for analyses using healthcare data sets