Well, there might be a fear that a Squib parent would increase the chance of more Squibs. Plus, just the sheer distain for someone who was born to a magic family, then doesn't come into any power.

Also, I think Filch is sensitive to the fact that he is a Squib being constantly surrounded by magic.