I remember how tense the atmosphere was living in a castle in the Welsh Marches. It meant long nights in a huge complex of dark, at times menacing stone buildings and long marches through the cold to get to important areas of the castle. I seem to remember a walk to a tower some distance from the keep which involved a very unnerving path across a narrow plank held up by timbers jutting from the wall. Seems it may have been to a garderobe but I’m not sure. And of course, I don’t need to talk about the unpleasantness of the medieval garderobe on a cold, rainy night.
Also, the walls were slick in places, especially in the many passageways inside the walls and keep. Only the few rooms with glass didn’t have patches of moss or slime mold or something unpleasant on them that fed on the huge amounts of condensation from our breath. As we ran through the hallways and passages in the dark some nights, we kept one hand on the wall so that we could find our way without needing to waste a candle during a full moon, and the cold dampness was always deeply unpleasant.
Tapestries and wall hangings were as much to brighten a room and keep out the chill as they were a means of wicking the abundant moisture away from the walls, though some interior rooms were dry enough not to need them for that purpose. And of course, the dampness only increased the further down the structure you went; that was one of the many reasons lords and ladies slept on the upper floors.
I don’t think I would want to live in a castle. Even a weatherproofed one would be too large and too full of empty space and inconveniently-located spaces over a wide area to be a practical or comfortable home. Unless you’ve got an enemy outside your gates, it’s just not worth the trouble.