Yes, it is a big deal, but probably not for the reason you are thinking. Sugar in your coffee.. Really? Do you drink vodka cranberries too?
I want you to start drinking black coffee and straight spirits for a while. Eventually you can add a little cream to your coffee or an ice cube to your whiskey.
Good riddance.