The lands where an ancient fortress once stood were known as Colombier until the 17th century. Acquired in the early 16th century by Jean Le Breton, France's Controller-General for War under King Francis I, a new château was constructed around the original 14th-century keep where King Philip II of France once met Richard I of England to discuss peace. It is also known for its beautiful gardens.
The château remained in the Le Breton family for more than two centuries until it was acquired by the Marquis de Castellane. During the French Revolution the property was confiscated and in the early 19th century, Emperor Napoleon acquired it for his brother Joseph Bonaparte.