How do your colleagues feel about this? If they're not bothered, then you'll likely look a douche to them too if you make a fuss, no matter how justified. If they are all "damn lazy asshole not getting a substitute", then the whole lot of you should ask him.