Søborg Castle, in its heyday, was the strongest castle in Denmark, and was also used as a prison. It was inhabited until the Count's Feud in 1535, when it is speculated that it was destroyed. In 1577, the feudal tenant was granted permission to use the ruins as a quarry.