I think it's a blind spot among the wizard-raised. They rarely resort to physical violence, even when deprived of their wands -- in the Pensieve scene, Snape keeps going for his wand instead of just hitting James in the gut, which would have been just as effective.

The DEs keep casting curses at Harry and company at the Ministry, even though they were adults, outnumbered the kids, and were certainly large enough to just *manhandle* them.

The only fist-fight we've seen was Harry, Fred and George jumping Draco on the Quidditch pitch, and McGonagall broke that up with a dismissive comment about "Muggle dueling".

So maybe Voldemort and the DEs don't even think about simply picking up a knife and attacking when they're too tired to cast, though that would be the logical thing to do.