Patterson initially gained recognition for his exceptional skills at St. Vincent Pallotti High School in Laurel, Maryland. During his high school career, he rushed for over 4,000 yards and 55 touchdowns, earning several accolades and honors. This success helped him secure a scholarship to play college football at the University at Buffalo.