The first castle on the site was built in the 11th century and probably consisted of a motte with wooden fortifications.
In 1281 Stephen of Penchester was granted a licence to crenellate by Edward I and built a stone castle. The oldest part of the castle shows a distinctive herringbone pattern In the 15th century the castle was converted into a fortified mansion.
Although the castle was never attacked or besieged a major fire in around 1600 left only enough of the castle to make a farmhouse.
In the early part of the 20th century the castle was restored by the explorer Sir Martin Conway.
In 1951 following the comprehensive restoration the site was taken over by the Carmelite order who do not open the castle to the public.