The reason I think this is a fold pre flop is because we can't win massive pots. We can easily lose our stack with top pair, and the flushes we make are not enough compensation for the times we are dominated. When we're full stacked, it's not so easy to lose your stack with a bad top pair, but you do increase the potential size of the pots you win when your flush comes good.
We do have great pot odds, but this is about implied odds vs reverse implied odds.