Colours are fine for the game itself, but if you plan on going out afterwards it's best to go neutral as a lot of places will have policies about football colours.
A scarf is the sort of thing you can put away in a bag or whatever so I imagine you won't go far wrong with that.