Though that isn't entirely the Dursley's doing; Harry was with his real parents for roughly the first year of his life. At least, I think he was. I'm not very good with math. :/