Road safety is a major issue in every society. The assessment of driving ability with a real vehicle is a lengthy and costly process; therefore, there is a growing need for the development of a neuropsychological battery that can provide a fast and reliable evaluation of a person’s cognitive fitness to drive. In the present study, we examined the relationship of an off-road lab-type test, namely, the Driving Scenes test, with performance on a driving simulator, as well as the influence of cognitive factors on driving ability as evaluated by Driving Scenes. Our results demonstrated a relationship between Driving Scenes and driving simulator performance. They also showed that some cognitive factors (namely, selective attention and verbal memory), were predictive of driving ability (as determined by the Driving Scenes test), but not others (namely visuospatial perception/memory, working memory, and visuospatial recognition). In addition, age strongly predicted performance on this test (younger age was associated with better performance). The conclusions derived from the present study highlight the need to identify off-road tools with high predictive value in assessing driving ability.