As per the first link here, its to the right of the wall that the gate is by. Or at least by my interpretation of old maps.
Thanks so much for your ideas! My question was more "is there a way to officially access the ruins", but honestly now you've pointed these bits out to me climbing seems more fun...