Photo Charles Taylor

The first manor house was built on the site in the 11th century. It was rebuilt in 1441 using brick construction after Sir Roger Fiennes was granted a licence to crenellate. It was abandoned at the end of the 18th century and all the contents and much of the interior structure was sold off in 1776. The ruins were then restored between 1913 and 1933.

From 1957 to 1988 its grounds were the home of the Royal Observatory, Greenwich. It is now the International Study Center of Queen's University, Canada.

The gardens are open April to October and there are some guided tours of the inside of the castle (see their website).