Ayton Castle is a late 14th century stone pele tower, three storeys high. It was partially demolished in the 17th century to provide building stone and has been in ruins since.