I think it's just implying that Hermione spends more time in the library than out running about.
I agree. I mean, at Hermione's age, I took regular exercise, was quite fit, but ask me to run 100 yards with a group of people and I'd arrive long after them, wheezing away.

Plus, of course, there seems to be no PE at Hogwarts.