I think it just depends on how tolerant you are towards variance. I usually stay unless I have a 3 buy in+ stack and there is a good player with as much money at the table. If I have position on him I often stay anyway. If you think you may go on tilt if you lose too much money in one hand it would probably be better to leave though.