I think it's more likely he has A5, A6, AQ here than a set.

Anyways, i don't think you should fold here when you almost have expressed odds, your in position and your practically guaranteed the rest of his stack if you hit (and sometimes you'll get stacked, but it's probably still worth it).