While driver distraction remains an issue in modernized societies, technological advancements in data collection, storage and analysis provide the means for deeper insights of this complex phenomenon. In this research, factors influencing when driver distraction through mobile phone use occurs during naturalistic driving are investigated. Naturalistic data from a 6-stage, 230-driver experiment are exploited, in which drivers installed a non-intrusive driving recording application in their devices and conducted their trips normally across a 21-month timespan, coupled with corresponding questionnaire data. The various experiment stages involved providing progressively more behavioral feedback to drivers while continuing to record them. Subsequently, supervised Machine Learning XGBoost algorithms were employed to model the contributions of naturalistic driving and questionnaire features to the decision to engage mobile phone use. Mobile phone use percentages were heavily skewed towards zero, therefore imbalanced ML with a minority-oversampling approach in a binary format was employed. To increase the explainability offered by the algorithm, SHAP values were calculated for the informative features. Results indicate that the decision of drivers to use a mobile while driving is governed by a number of complex, non-linear relationships. Total trip distance is the most significant predictor variable by a wide margin, with mean SHAP values of 0.79 towards affecting the model decisions for the probability of mobile phone use of each driver. However, other variables influence the final predictions as well, such as the number of tickets in the last three years (m.SHAP = 0.30), declared mobile phone use (m.SHAP = 0.26), the amount and variety of provided feedback (m.SHAP = 0.17) (i.e. experiment phase number) and family member numbers (m.SHAP = 0.09) decrease the probability of using a mobile phone while driving. Conversely, increases in driver experience (m.SHAP = 0.22), driver age (m.SHAP = 0.11), engine capacity (m.SHAP = 0.11) and total kilometers driven annually (m.SHAP = 0.08) increase the probability of using a mobile phone in naturalistic driving conditions. SHAP dependency plots reveal non-linear effects present in almost all variables. Fuel consumption had a particularly strong non-linear effect, as higher values of this variable lead to both higher and lower probability of drivers using a mobile phone, deviating from the safer average. Legislation, campaigns and enforcement measures can be restructured to take advantage of gains margins in terms of understanding and predicting driver distraction behavior, as explored in the present study.