Some places already have gender neutral restrooms and haven't had too many issues.  Some people might think something along the lines of what would prevent a man raping a woman in a public restroom if restrooms were gender neutral - the same thing that's preventing a potential rapist from walking into a women's public restroom.  The little plastic sign on the door doesn't stop someone who is considering committing a violent act.

Next week, I will be attending the National Gay and Lesbian Task Force's Creating Change Conference in Detroit.  The public restrooms will be designated as gender neutral during the conference.  I haven't heard of any problems in the 3 conferences I've attended.