I think most commenters here are missing the point.
There is a more extreme reaction to transgender people as opposed to gay or lesbian people, because of issues like sports and bathrooms. And that hits at people's sense of injustice. For example if you have a young daughter, a lot of people will hate the idea of a person with a penis going into the women's room and being around there little girl. Or if that daughter grows up and joins a sports team, the idea of somebody who is hormonally male and thus naturally more muscular competing against your daughter is unpleasant.
Put differently, I think a lot of people we now classify as 'transphobic' don't actually have much problem with trans people themselves. Rather, with how the efforts to ensure trans people receive the full treatment of their chosen gender can affect the rest of society.
For me personally, I don't know what the answer is. I generally don't care which bathroom you use as long as you wash your hands. I have no problem with anyone presenting themselves to the world as whatever they wish, if it makes you happier than by all means.
At the same time though, I don't think it's transphobic to point out that somebody who is largely or entirely biologically male will have a natural competitive advantage in the field of sports.
So while I certainly don't want to exclude anybody, I think there is at least a little justification for restricting some women's sports to those who are genetically female.