I could not care less who else uses the same bathroom I do. A lot of other people do care, though, so it's become a political and moral issue.
The most well-publicized example of bathroom control is North Carolina's HB2 law. The bit of the law that has gotten the most attention is the restriction that people must only use bathrooms corresponding to the sex on their birth certificate. That's seen as an attack on transgender people, which I'm sure is exactly the intent, even if the politicians involved say otherwise.
So why do they say the HB2 restrictions were necessary? The main arguments seem to be privacy and security. That doesn't make any sense, though. I'm just as much at risk of some other man invading my privacy or assaulting me in a bathroom as I would be with a transgender woman using the same bathroom. You'd be better off banning cell phones and cameras in bathrooms, if privacy is your main concern. (And good luck enforcing that...separating a modern American from his cell phone is a major undertaking.) As for security, sexual predators won't let a little thing like state law stop them if they're intent on causing harm. By definition, those people are lawbreakers. Besides, there are already plenty of laws against such predators that still hold regardless of how transgender people are treated. The privacy and security arguments for this law just don't hold up.
Neither side of this problem gets much sympathy from me on this specific issue. I think it's ridiculous that anyone cares who uses which bathroom. Men, women, horses, Klingons...as long as they mind their own business and leave me to mine, who cares? I'd be perfectly happy if every public building just installed one big unisex facility with a bunch of stalls and a shared set of sinks. (And paper towel dispensers. Those hand-dryer things just don't work as well.)
However, I do have a problem when laws are enacted to target specific groups of people. I don't understand transgender people. My head just doesn't fit around the idea that you'd be so unhappy and uncomfortable that you'd want to make such a fundamental change to yourself (both mentally and physically). But they're still people, who deserve the right to do wherever they want as long as no one gets hurt.
The NCAA seems to agree, judging from the way they've moved a bunch of college sports events out of North Carolina. The reason given was that North Carolina's laws don't "promote an inclusive atmosphere for all college athletes, coaches, administrators and fans." The ACC is following suit. College sports is a big deal in North Carolina, both in terms of popularity and financial impact. Whether it's big enough to convince the public to pressure their elected officials into a more tolerant stance remains to be seen.
I'd love to see this issue just completely disappear because North Carolina legislators realize how stupid it is, but that seems highly unlikely. The next best thing would be for the public in North Carolina to make their displeasure known at the polls. They'll get their chance in November, but that also seems unlikely. So we may be left with protests and actions like what the NCAA and ACC have taken. Not the best scenario, but it may be the only feasible one.