It's really up to your landlord in a legal sense - they are effectively stealing land from him. But in theory there is nothing stopping you tearing down the fence apart from the agro it will cause. They after all have no right to build the fence in the first place.