Generally speaking I don't think it is fair, but in this case I don't think their appearance is entirely irrelevant.
These people go out of their way to make themselves as unattractive to ordinary people as they possibly can. The question is why? And one answer might be that they make themselves look horrible to make other people hate them - thus making it all the easier for them to hate other people!
This picture has made the rounds but I just saw it again on Facebook where it became the subject of a discussion as to whether, no matter how much we may disapprove or even dislike some people, it's fair to use their appearance against them.