Some will have ID badges and they should have the charity number which you can check on OSCR.
If you're concerned best bet is contacting the charity and giving description/location of collectors. If they are unaware and you have a suspicion of fraud it becomes a police matter.