Yes, he really said that. He also said in a university enrollment of 20,000 undergraduates, he wasn’t going to spend much, if any, time on the special needs of just 400-ish athletes.
(It was not part of how he was measured by the BOR nor how his compensation was determined.)