I think that's often the same argument in different words. "Your offense is unwarranted" = "My actions were not offensive" = "I didn't intend for my actions to be offensive". They're still defining what is offensive using intention as the criterion, rather than effect.

Sometimes they mean instead, "I know other PWDs who aren't offended by this so it isn't offensive", which is a slightly different argument. They're accepting effect as a criterion for determining what is acceptable, but they're carefully cherry-picking which effects to consider. If I slap you in the face, the vast majority of people on Earth are not hurt or even affected by that. How do you decide which effect is the valid or correct one?

If they mean "I don't care if everyone in the world were offended by this, they'd still be wrong", well, there's not much you can say to that. Some people think the world is full of folks who "just want to be offended" and "need to grow a thicker skin". In my experience, this is probably not the case. It is interesting, though, that there are majoritarians who think the minority (of whatever stripe) is conspiring to control their words and actions, and that really pisses them off.