Why not say that the first time then. It's a valid argument.
Why turn the words around so they mean something completely absurd?
If you say 'that's irrelevant', then leave it at that, you're doing fine.
it's when you say 'are you saying xyz?' , where 'xyz' is obviously retarded, when the person obviously isn't saying that, then you're being an idiot.