You're right and wrong. Statistically speaking, there should be ~3.5 diamonds among the hands of the rest of the players. You can't assume that when making your calculations though.