I do this to players on my left a lot. Ramping up your aggression even further and starting to 4-bet me will stop me from doing this to you with not much but then I just tighten up and keep at you with real hands. If I'm doing it with not a lot in the first place you're usually giving off a bet size tell of some sort. You can close that hole by being consistent or sucking me in by underbetting real hands once in a while.

Often it comes down to who leaves the table first. I never want you on my left but if you're there I'll play and try to frustrate you.