To be honest - unless you’re swimming in a structured club environment where everyone is in lane according to ability and following the same timed sets you’ll always have some issues with people getting in the way.
Even triathletes when I used to compete would join the club for a short while and really struggle to not get in the way.
Thankfully now I am much fatter so have less of these issues