Do you agree that someone can theoretically have a legal right to do something bad (as in, be legally allowed to do it) without that being a good or moral right for them to have?
I think you're only believing "right" to mean one thing and one thing only, when I'm using it in a sense where legality and morality don't necessarily coincide (even if they do in other contexts, conditionally).
So when I say they had the legal right to own slaves, and that right was taken away from them, that isn't a matter of opinion/belief because that's factually what happened, but that doesn't mean that I think they had the right morally speaking, which is a different concept.
I hope this makes sense.
If it was legal for certain people to slap certain other people, then the people doing the slapping would have the authority over the people being slapped to slap them. But then if the law was changed and took away their authority to slap them, that would be using authority over those slappers to stop them. Does this make sense? Both can be true at the same time