It's not your mask, everyone has seen people wearing masks, it's not remotely noteworthy. Ok, there's not as many people wearing them that should be, but there's still plenty of them around.
People are just nervous in public at the moment, that's all
Anyway, even if it is making people avoid you - good thing, no?