OP did say “should I ask if they mind”, which is de facto asking for permission.
If I wanted to be polite, I would go around and inform them that I was going to use the garden. That way you’ve been considerate enough to do that, but haven’t asked permission (because of the problems stated above). That way they can raise concerns but you haven’t given them the right of veto by asking.