One of the problems I have with it is that Grawp is presented as mentally deficient, and it takes some analysis to arrive at the idea that it may just be him, and not his species. The most obvious reading is that giants are less than sentient, IMO, and then Hagrid isn't necessarily stupid but he is uneducated, so you get to wondering exactly what JKR is trying to say.

Fleur's a quarter Veela, and that doesn't seem to bother anyone, so it isn't cross-species breeding per se that's the problem. It's something else.