A downswing is part of regular swings. The fact that you've gotten unlucky in the past does not affect your chances of being unlucky in the future.

Move down when you no longer feel like you have an edge, or you are below 20 buy-ins (or 10 or whatever)