The empty castle burned at some point after the family left, and then it was quarried. In 1833 its iron fittings were pulled out and carted down to build a sugar factory — the ruin was worth more as material than as a monument, which is exactly how people saw old castles before heritage protection existed. By the middle of the nineteenth century the estate had given the hill up for good, and the forest began to close over the walls. The end of Gýmeš was not a siege. It was out-lived by a warmer house in the valley, and by a century that had other uses for its iron.Prázdny hrad po odchode rodu vyhorel a potom sa z neho stal lom. V roku 1833 z neho vytrhali železné konštrukcie a odviezli ich na stavbu cukrovaru — zrúcanina mala väčšiu cenu ako materiál než ako pamiatka, čo presne vystihuje, ako sa na staré hrady hľadelo pred vznikom pamiatkovej ochrany. V polovici 19. storočia panstvo hrad definitívne opustilo a les sa začal zatvárať nad múrmi. Koniec Gýmeša nepriniesol útok. Prehral s teplejším domom v doline a so storočím, ktoré malo pre jeho železo iné využitie.
