The castle's site is well suited for harbor defense, and appears to have been so used since the 15th century. According to the Record Office of Genoa, cannon batteries were constructed on the site in the early 16th century, and military engineer Gian Maria Olgiato drew up plans for a full fortress circa 1554. The resultant castello was completed by 1557, and in 1575 was instrumental in turning back an attack on the town by Giò Andrea Doria. From 1622 to 1624 the structure was enlarged, and in this form it survived for a century and a half, until a 1798 attack by the English (during Napoleon's reign) destroyed its little tower. The fort was then abandoned after the Congress of Vienna in 1815.
In 1867 the structure was purchased for 7,000 lire by Montague Yeats Brown, then English consul in Genoa. Under his direction the fort was transformed to a comfortable villa without substantial alteration in its general form. His descendants held the property until 1949, then sold it to an English couple, Colonel and Mrs. John Baber, who restored several ruined sections, until they in turn sold it in 1961 to the City of Portofino.