The castle on top of the hill can be seen from afar and dominates the town. It was built, together with the town walls, under the reign of king Diniz in the 13th century over the remains of a Roman castellum that had been fortified by the Moors. It consists of battlemented walls with four square corner towers and a central granite and marble keep (Torre de Menagem), with its height of 40 m the highest in Portugal. The top of the keep can be accessed via a spiral staircase with 197 steps, passing three stellar-vaulted rooms with Gothic windows. The merlons of the machicolation around the keep are topped with small pyramids. Standing on the battlements, one has a sensational panorama over the surrounding landscape. One can also glimpse the remains of the city walls that once had forty turrets and five gates. The castle now houses a small military museum.
The square in front of the castle is named after Gonçalo Mendes da Maia or O Lidador, a brave knight killed in the battle against the Moors in 1170.