This paper tries to identify and investigate the most significant factors that influenced the relationship between COVID-19 pandemic metrics (i.e., COVID-19 cases, fatalities, and reproduction rate) and restrictions (i.e., stringency index and lockdown measures) with driving behavior in the entire year 2020. To that aim, naturalistic driving data for a 12-month timeframe were exploited and analyzed. The examined driving behavior variables included harsh acceleration and harsh braking event rates concerning the time period before, during, and after the lockdown measures in Greece. The harsh event rates were extracted using data obtained by a specially developed smartphone application which were transmitted to a backend telematics platform between the 1st of January and the 31st of December, 2020. Based on the collected data, XGBoost feature analysis algorithms were deployed to obtain the most significant factors. Furthermore, a comparison among the first COVID-19 lockdown (i.e., March–May 2020), the second one (i.e., November–December 2020), and the period without COVID-19 restrictions (i.e., January–March and May–November 2020) was drawn. COVID-19 new cases and new fatalities were the most significant factors related to COVID-19 metrics impacting driving behavior. Additionally, the correlation between driving behavior with other factors (i.e., distance traveled, mobile use, driving requests, and driving during risky hours) was revealed. Furthermore, the differences and similarities of the harsh event rates between the two lockdown periods were identified. This paper tries to fill this gap in the existing literature concerning a feature analysis for the entire 2020 and including the first and second lockdown restrictions of the COVID-19 pandemic in Greece.