It was built at the end of the 12th century as the main residence of the Princes of South Wales. It sits high up on a hill overlooking the River Tywi, with ditches cut from solid rock to form a moat as an extra protection on the less defended sides. The round tower and much of the walls date from the early 13th century. It was abandoned at the end of the 15th century until the 17th century when it was partially restored for use as a summer house. This is when the structure on top of the tower was added. It was finally left to decay after being badly damaged in the 18th century.