The purpose of this study was to explore the factors determining driving difficulties as seen from the viewpoint of thirty elderly MCI drivers and thirty age-matched controls without cognitive impairment, using data from an extensive questionnaire. The analysis revealed that two factors underlie MCI perceptions of driving difficulties, representing (i) difficulties associated with late detection combined with slowed response to relevant targets in the peripheral field of view, and (ii) difficulties associated with divided attention between tasks requiring switching from automatic to controlled responses, particularly of long duration. The analysis for healthy controls revealed three factors representing (i) difficulties in estimating speed and distance of approaching vehicles in complex (attention-dividing) high-information-load conditions; (ii) difficulties in moving head, neck and feet; and (iii) difficulties in switching from automatic to controlled processing in new or unexpected situations. The exposure pattern of both groups is generally consistent with their concerns. The results underline the ability of elderly drivers’ (with MCI and without cognitive impairment) to indicate probable impairments in various driving skills. The patterns of difficulties identified may provide us with indications about roadway situations that are associated with perceived demands by each group; these patterns might be important to consider when auditing or designing improvements to accommodate aging road users, as well as the most vulnerable subgroup of elderly drivers who might still be able to drive.