Why ostracise crossdressers?
Because ostracising others who are different makes you cool. While not everyone ostracizing crossdressers, most people find some group of people to shun. Immigrants, atheists, drug addicts, the upper class, gays, communists, the mentally ill/handicapped, the homeless, the military, the other team, the neighbouring state, some foreign country, police officers, another religious belief, etc.