These are good points. I also imagine Dumbledore isn't just going around using Legilimency on people without just cause. It would be a massive invasion of privacy, and he is, after all, ostensibly one of the good guys.