The first record of a fortification at Bishop Stortford was in the Anglo-Saxon Chronicles in AD 921. A Norman motte and bailey castle was then built on the site following the Conquest. Some rebuilding work was carried out on the orders of King John and the castle later came in to the ownershipo of the Bishops of London. The Bishops used Waytemore as a clerical prison for over 200 years.

Following the Civil War the castle was sold and the castle demolished when the new owners sold off the walls and other structures for building materials.

This has left little physical evidence of the castle apart from the Norman Motte.