Gardner began his youth career at Aston Villa, a club he joined at the age of 6. He spent over a decade at Villa, progressing through the academy ranks before making his first-team debut in 2011. During his time at Villa, Gardner also had loan spells at clubs such as Coventry City, Sheffield Wednesday, Brighton & Hove Albion, and Nottingham Forest.