Truthfully you need to stop worrying about what you can't control. I'd take his comment about the previous owners with a pinch of salt. Unless they sat around all day without moving then there would have been noise.
I think your neighbour needs to think about finding another flat without upstairs nieghbours.
Get on with living your life and eventually they'll tire of complaining.