Potting the pot may rarely be a bluff, but it's also often a bet from a player happy to take it down. A9s TTs T9s etc. Seriously he's played 4 of the 20 hands you've played with him. This is not large enough to put him on a set.