The original fabric of the castle dates from the 14th century. It consisted then of a small keep set upon the south wall of a great courtyard of oblong shape, with the keep projecting beyond the walls to the South.
Previously to the 14th century the Barony of Ballinbreich was held by the ancient family of Abernethy but ten passed through marriage into the hands of the Earls of Rothes.
In the sixteenth century alterations and additions were made that gave the castle the outline that can still be seen today. It has been mined for generations and it was when one of the inner walls collapsed that the 14th century masonry work was revealed. The workmanship of the masonry is considered unsurpassed in Scotland.
Ballinbreich is situated on private ground and anyone wishing to visit should first gain permission.