Misleading title.

~~If my thing was public in the past, and I took it private, the old public code is still public.~~

That's... How the Internet works anyway.

Edit: See Eager Eagle's better explanation below.

TL;DR - be careful who you allow to fork your private repos. And if you need to take a public repo, which has forks, private, consider archiving the repo and doing all the new work in a new repo. Which is arguably the reasonable thing to do anyway.

Still a misleading title. This isn't a way to break into all or even most of your private repositories.