I have lived at Sissinghurst, on and off, for the past 45 years. For my entire conscious life it has been what I have thought of as home, even when living away, in London or abroad. I still live with my family now in the southern half of the long pink brick front range, built in the 16th century, swathed in roses, part of an enormous palace, most of which has long disappeared.

My father, Nigel Nicolson, adapted it in the early 1960s from two cottages. My mother, Philippa, installed a new, clean pine kitchen, with dark-blue Formica surfaces and a big six-burner gas stove in the centre of it. There she cooked enormous cartwheels of mushroom quiche, sole in creamy sauces, giant pieces of roast beef, and roast potatoes that she fried to make crisp.

The dining room was lit with silver sconces that my grandmother, the writer Vita Sackville-West, had brought to Sissinghurst from Knole, her own childhood home in Sevenoaks, Kent. They were fixed to the umber hessian of the walls, and between them hung a large portrait of a Sackville ancestor, the eloquent 1stEarl of Dorset, who had been chosen by Queen Elizabeth I to give Mary, Queen of Scots news of her impending execution. His rheumy, pink-edged eyes followed me around the room. Beside him were two pictures by John Piper, of Knole in a rainstorm and of a snake strangling ahorse, somehow made out of sand.

There was a carved chest, inlaid with mother-of-pearl, to hold the glasses and one or two odd framed things: a drawing on a menu from a London restaurant in 1919, Edwin Lutyens's first sketch of the Cenotaph, made for Vita over lunch; a poem of Verlaine's written in high calligraphy on vellum, Je suis venu, calme orphelin, an offering to Vita from one of her lovers; and a battered medieval wooden Spanish saint called Barbara, the first present to her from her husband (Sir Harold Nicolson, my grandfather) when firstin love. The place was full of fragments salvaged from earlier lives.

The castle garden, which was created by my grandmother and grandfather, then wasn't treated with the sense of exquisite preciousness that it is today. My friends and I used to bike around it and invent racetracks through it. The start was at the front of the house, between the 19th-century bronze urns and the floppy arms of the old rosemary bushes: down through the medieval gateway into the upper courtyard, sharp right in front of the pink brick Elizabethan tower swathed in clematis, across the lawn and through the narrow gateway into the Rose Garden - a terrifyingly spiny acanthus bush on the corner there - a zigzag through the alliums and the mounds of old roses before hitting the fastest of all straights through the Lime Walk and under the branching hazels of the Nuttery, where I would finally skid to a halt outside the Herb Garden. The whole thing, which is slightly downhill, could be done in just under a minute.

There were no exclusions. Our dog, Rip, a terrier, chased cats down the yew walk and through the Rose Garden. In the evenings my father used to stand on the lawns with a tennis racket and hit a ball as high as he could - 60, 80, 100ft in the air - and ask one or other of us to catch it as it came down, stinging our fingers or smacking our palms, all for increasing double-or-quits money. We did three-legged races from one end of the courtyards to another. And it wasn't only us: children from the village used to come to play hide-and-seek in the garden. All the gates were always open. There was scarcely a lock on the place. It would be difficult to imagine a more wonderful environment in which to grow up.

Since 1967 Sissinghurst has belonged to the National Trust. My father handed it over as a way of paying the death duties on his mother's estate, with the proviso that his family could live there for ever. It is a slightly strange and anomalous role: we are tenants in what was our own house. The term that the trust uses is “resident donor”. Even the phrase reveals the awkwardness, as if one were living in a body whose heart has been given away. It's your home but someone else's business, a place that is yours emotionally but very far from being yours in terms of who owns it, runs it, pays for it, works at it and conceives of its future.

It doesn't take much to imagine this relationship going wrong, but it is also perfectly possible to make it go right. We all carry the mutual-respect agenda firmly in the front of our minds. It is a plain fact that we would not be there unless the trust were. We are now in the middle of an exciting transformation of the farm and restaurant at Sissinghurst, with the aim of feeding visitors - more than 100,000 a year - with produce grown on the farm. It was my idea originally and to its credit the trust is backing it to the full. And if I step away from the daily wrangles and frustrations of making any complex organisation work, I can only think what wonderful luck I have had in being born attached to - you might even say bound to - this beautiful and extraordinary place.

Adam Nicolson is the author of Sissinghurst: An Unfinished History

Explore Property

• Buying & Selling

• New Homes

• Overseas

• Interiors

• Investment

• Gardens

• Cool In Your Code

• Specials

• First-time buyer

Times recommends

• Second homes: credit crunch casualty or money-spinner?
• How I built my own sliding house
• Top 10 incentives for house buyers