Can you explain, in simple terms, how orbit works and how it is stable?

As I understand it, the earth is in a constant state of falling toward the sun and "sling-shotting" around it, but my question is why is there a lateral force causing it to move transversally to the sun instead of falling perpendicularly straight down?