Professor McGonagall: Albus, do you really think it safe, leaving him with these people? I've watched them all day. They're the worst sort of Muggles imaginable. They really are... Dumbledore: The only family he has. Professor McGonagall: This boy will be famous. There won't be a child in our world who doesn't know his name. Dumbledore: Exactly. He's far better off growing up away from all of that. Until he is ready.
The professors leave baby Harry with his aunt and uncle.