I recall them saying things like that, but it doesn't make too much sense. Why was a Stark of Winterfell raised by the Warden of the East? And why also Robert Baratheon? It doesn't make much sense when the authors suggest some strong cultural and regional histories of the characters, but then also say they were super close buddies

The storytelling has these characters in two places at once, or at least that's the impression of it. It certainly appears that GRRM has some logistics problems. One example would be that the Greyjoys are any sort of threat to large inland nations