I've been attacked before because I was wearing clothes with colours corresponding to one of the teams (not going to say which because I don't want to fuel that debate) by supporters of the other team whilst I was walking through town.
I could not give the tiniest shite about football.