Are you young? Were you going far from the city centre?
The drivers want short trips close to town not close to the boundary. I'd have took a note of his number and reported his behaviour black cab is still your safest bet for transport that time of night but you may need to be a bit assertive regarding not handing over money and ensuring the meter starts at the base rate.