The more I think about it, the more it becomes obvious that Vernon is indeed very bullied by Marge. When Harry suggests that he might let something slip to Marge, Vernon's reaction isn't just anger, but "panic". Admittedly, he stresses about being perceived as normal in general, but being "outed" to his sister is panic-inducing.

Mm... I feel the potential for Dursleyfic. *g*