The original castle was started in 1101 by the River Severn and enlarged throughout the 12th century. The keep was probably built in the early 12th century. The castle was besiged in the Civil War and after it was captured it was destroyed apart from the keep which although undermined still stands but now leans dramatically. Nearby is Panpudding Hill which was used to beseige Bridgenorth Castle in the 12th and 17th centuries.
The town was also surrounded by a turf rampart in the 13th century, later construced in stone with five gates, but only one gate, the North Gate, now remains.