With the flush draw there I don't blame you for raising, but you'r heads up against the preflop 3b-er, so unless it was a squeeze attempt with some suited connectors or suited ace, which I guess it could be, the only draws he's left with are AKh or maybe AQh. If you had an overpair in this situation I'd always raise with the flush draw there, but with a set even if the flush completes you have redraws to a fh so I'm less concerned.