The area about Green Point was initially used by the Dutch as a cattle grazing station. After the British annexed the Cape in 1806, they began to hold horse races on its common, and the area developed as one of Cape Town's most popular social centres. William Burchell commented in 1811 that:
"Vehicles of every description, from the elegant London-built carriage of the Governor and the English curricle to the antiquated Dutch calash and the light but jolting paardewagon, are seen driving about to enjoy the sport ... Together with the art of making horses run fast the science and mystery of betting has found its way to the farthest extremity of Africa, and on Green Point large sums are said to have been won or lost."
In 1830 Green Point had 40 residents, among them Robert Crozier, the Postmaster General. In 1820 the construction of a lighthouse at Green Point, the first of its kind in southern Africa, was put forward by Sir Rufane Donkin. The project was designed by Hermann Schutte, a German stone mason who had previously been employed by the VOC, and was now working for the new administration as Inspector of Government Buildings. The lighthouse was commissioned in April 1824. Unfortunately its structure did not stand up well to the hardships of Cape weather, and by 1840 Governor Sir George Napier was describing it as "an old ruinous building, falling fast into decay". The building was rennovated in 1842, and in 1863 its height was increased by the addition of a new tower.
During the South African War of 1899-1902, Green Point common was used by the Military as an encampment for a large number of British troops as well as a camp for Boer POWs.