When the Dutch first landed at the Cape on 6 April 1652, under the leadership of Jan van Riebeeck. They erected a Fort built of sods on a site in the Table Valley, near the Beach and East of what is today Adderley Street. The structure, known as the Fort of Good Hope.
This was abandoned in 1665 when construction began on a new and more substantial building further east that would be able to resist any attack from the sea. It followed a standard plan common in Europe at that time, and was pentagonal with a bastion at each corner, a shape it retains to the present day.
Building of the Fort was entrusted to engineer Pieter Dombaer, assisted by carpenter Adriaan van Braeckel, and master mason Douwe Steyn. Four cornerstones were laid on 2 January 1666, and it was completed on 26 April 1679 when the old fort was finally evacuated. The five bastions were named after the Prince of Orange and contained the quarters of some of the officers and men. On top of each were mounted cannon of various sizes and, in time, the Castle, as it became known, was defended by about 100 pieces. Access was gained through a sally port with iron doors situated in the curtain wall between the Buren and Katzellenbogen bastions, facing the sea, but in 1682 this was moved between the Buren and the Leerdam.
Facilities within the Castle included the Governor's residence and a large Council Hall completed in 1695, which, until 1704, was also used as the first DRC Church at the Cape. Entrance to the Governor's residence was gained through a ponderous and much decorated stoep, jointly designed by sculptor Anreith and architect Thibault. Despite its show of might, the Castle is not known to have fired a shot in anger, and is one of a number of fortifications subsequently added to cover approaches to the Bay. This did not prevent the British from landing troops at Simonstown in 1796, and what meagre resistance was put up by the Dutch at the time took place further inland. It was declared a National Monument under old NMC legislation on 17 April 1936. It should also be remembered that this is not the first such building to have been erected at the Cape and that at least one fortification was erected close to its site before the Dutch ever landed.