Drivers with brain disorders have difficulties in their driving competence (motor, visual, cognitive or perceptual) which may lead to increased accident probability. Alongside, driver distraction is an important cause of vehicle accidents. The interaction of brain disorders and driver distraction, which has not been adequately investigated so far, makes the assessment of their driving competence a very challenging task. The objective of this paper is the analysis and quantification of the effect of mobile phone use on the driving competence of patients with neurological diseases affecting cognition diseases, through a large driving simulator experiment carried out by an interdisciplinary research team of civil engineers, neurologists and psychologists. 34 Controls, 43 MCI, 28 AD, and 20 PD patients, all older than 55 years old, were asked to drive in urban and rural driving simulated environment and under three distraction conditions: no distraction, conversation with a co-passenger and conversation through handheld mobile phone. Their mean speed, their reaction time and accident probability in unexpected incidents were under investigation. The regression analyses (24 generalized linear models) indicated several interesting results. The findings extracted from the patient groups’ regression analyses highlighted the detrimental impact of the mobile phone use on their reaction time and accident probability.