You open your door to tell them to piss off, but you're scared to leave your flat out of fear of having to walk past them? If I was you, I'd not answer the buzzer, or open the door. If they are buzzing that much and harassing you, record it. What's your reasoning for not wanting to call the police?