Meanings and connotations of words are agreed on by committee. So if society agrees that a word has a particular meaning, a particular connotation, and that these both depend on context, then that's the way it is. You can disagree with it or not care about it, but it's not your call to decide whether a word is offensive or not to other people.

So if you call a random gay person 'faggot' or a random black person 'nigger' in today's society and they get offended, you're the asshole, not them, because you understood the word was highly likely to be offensive to them and you used it anyways. By the very fact of using the word, with full awareness of its meaning and connotation, you've intended to be offensive. You can't switch the responsibility on the listener for not understanding that you're not an asshole, you're just ignorant for thinking they shouldn't be offended by your being offensive.

If you call your friend one of those words, and the two of you agree it's inoffensive in that context, then you're not an asshole.

You can also not be an asshole if you use a word without knowing it has a negative connotation. But if the listener is offended by that word, that doesn't make them an asshole either. It's just a misunderstanding.