Cognitive functions which decline over age are of critical importance regarding driving performance. Neurological diseases affecting a person’s brain functioning, may significantly deteriorate the person’s driving competence, especially when unexpected incidents occur. What appears to be missing from the previous research, is the evaluation of driving behavior by using multiple driving indexes in a combined integrated manner instead of using single measures that focus on a sub-area of driving performance. The objective of the present study is to fill in this gap, mathematically, by latent analysis techniques, analyzing the traffic and safety behavior of drivers with neurological diseases affecting cognitive functions. More specifically, the impact of brain pathologies and several other risk factors, on reaction time, accident probability, and driving performance is under investigation. The neurological diseases affecting cognitive functions concern Alzheimer’s disease (AD), Parkinson’s disease (PD), and Mild Cognitive Impairment (MCI). A large-scale driving simulator experiment was carried out, comprising a medical/neurological and neuropsychological assessment of 225 active drivers, and a set of driving tasks for different traffic volumes, different driving environments, including in-vehicle distraction conditions. The statistical analysis methodology developed and implemented was based on Principal Component Analysis and Structural Equation Models (SEMs). SEM results indicated that the impact of neurological diseases affecting cognitive functions is significantly detrimental on the latent variables “driving performance” and on the observed variables “reaction time” and “accident probability”. The AD group had the worse driving behavior profile among the examined groups with neurological diseases affecting cognitive functions.