The Black Watch at the Battle of the Alma on 20 September 1854 during the Crimean War. When war broke out, authorities feared that Russian warships would attack the Australian colonies. This lead to the fortification of Pinchgut Island in Sydney Harbour.

The fort was renamed Fort Denison after the Governor of New South Wales at the time, Sir William Denison. The fort was actually completed after the Crimean War had ended.