Phillips began his football career at Archer High School in Lawrenceville, where he quickly gained recognition for his exceptional skills on the field. After graduating high school, he went on to play college football at Clemson University, where he continued to excel as a safety for the Tigers. During his time at Clemson, Phillips played in 37 games, recording 74 tackles, two interceptions, and two fumble recoveries.