Well, mint.com it is!

My bank (credit union actually) doesn't offer a limited access option like celtic123 mentioned, but they did say that if my account is compromised by mint.com I'd be covered under their CUNA insurance.

It would still be a (unlikely) headache to deal with if something went bad, but at least it would just be a short-term headache.

Thanks all!