You were being made fun of but it's nothing to do with racism.
I've had people shout "wanker" when I'm out jogging just minding my own business. People like to pick on joggers I think as if you're out getting fit it makes others feel worse about themselves.