If you are scared - which it sounds like you are (and reasonably so since it is a big chunk of your roll) - I think it is best to leave the table.

If you are playing balls out - then the other big stack is also giving you a chance to basically double your roll. So that's something to think about as well.