why do people say they do things "behind people's backs"?

Surely, if you are behind someone's back, you are actually in front of them?