They say poker is a game of incomplete information.

For the individual playing poker, this makes it a game of estimation.

Theory of Poker goes into some really basic examples about how I spot an opponent making an individual mistake, judging by how frequently he is given cards that allow him to make that mistake and how much it costs him every time I can calculate that he's leaving $18 on the table every hour. I cannot be guaranteed to be the person in the hand so the $18 is split between the 9 other people on the table giving me an average $2 profit per hour from sitting at this table with this opponent.

On a larger scale you keep an eye on all the players at the table - note individually how much better or worse they are than you based on how they play, factor in the type of mistakes they make (folding too much is often a less costly mistake than being a maniac) and what amount it's worth, factor in position, adjust your strategy to play more hands against the worst opponents that you have position on and estimate based on the above how much you are winning or losing per hour - and if that amount beats the rake you stay at the table.

There is no simple calculation that returns your edge at a table - it is an estimation exercise and the above is a simplified explanation of what you are trying to accomplish by it.