I want to get rid of cable, but would miss watching sports too much. It still feels somewhat wasteful.

As Bikes said, your download speed doesn't really matter because most sites can only provide so much bandwidth to you. It's been awhile since I checked, but getting over 500k on one download was about tops.