Most people walking about in Glasgow wearing green, blue or orange probably haven't had the football connotations even cross their mind. You'll be absolutely fine.
Of course there are certain pubs where wearing the wrong colour might be seen as a provocation but you're unlikely to wander into one of those without knowing what you're doing.