“PLACED amid beautiful scenery, which at this joyous season is clothed with the flowers and tender green of early summer, is the residence of Mr George Drummond. It is near to the hamlet of Penshurst, and is approached . . . by lanes that run over hill and through dale, with here and there rose-covered cottages set in apple orchards pink with bloom.” So frothed Country Life magazine in 1906, about Swaylands, a grand country mansion in Kent and the setting for many a fashionable society party.

A year ago Swaylands was almost derelict. Now the decline has been arrested by a development company, Oakdene Homes, which is converting it into luxury flats.

The history of Swaylands has been documented by a local historian, Marianne Thorne. The house was built in 1842 by a solicitor, William Woodgate, but it was only after its purchase in 1877 by London banker George Drummond, that it came into its own. Drummond made Swaylands a showcase for his wealth and influence. Local papers proudly recorded social visits by Edward VII and his daughter-in-law, later to be Queen Mary. A nursemaid at Swaylands, the incomparably named Ida Hogflesh, told Thorne that during the famous house parties maids were told to leave their bedroom doors unlocked in case visiting gentlemen wanted to avail themselves.

Drummond also had an eyefor the big statement, expanding the estate to 900 acres and building what was then the largest rock garden in the world. Vast sandstone boulders were transported by horse and cart to cover an area of five acres; river water was pumped up to create waterfalls. Ida Hogflesh’s daily duties included pushing babies round the rockery in a scarlet pram embossed with a large golden D.

After Drummond’s death in 1917, his son sold Swaylands to Sir Ernest Cassel, Edward VII’s financial advisor, who turned it into a hospital for soldiers. After the Second World War it was sold to Middlesex County Council for use as a school for boys with learning difficulties, a function it retained until 1994. It was bought by a developer, Honeygrove, in 2000, which was in turn bought by Oakdene Homes for £9.4 million in 2005.

Oakdene’s application to convert Swaylands into flats was opposed by locals, who feared “the enclosed millionaires’ homes would be like Beverly Hills”, said the Kent & Sussex Courier. Sue Massingham, sales and marketing director at Oakdene, says local people were in fact “amazed at the prices we were planning to charge, and were reassured that this would increase the value of their own properties”. Planning permission for 28 flats plus two new wings, of ten flats each,was duly granted, and work began early last year. All flats will have two or three bedrooms and range from 1,260 sq ft to 3,635 sq ft, and in price from £535,000 to £1.5 million. Buyers must stump up £15,000 for a garage and an annual £3,000, on average. for maintenance.

Oakdene is restoring two lakes and a sunken garden, formerly the school swimming pool. Residents will also have use of a tennis court, running track and cricket pitch, as well as the rockery, under the care of an estate manager. The rockery today is all bare boulders split by mighty tree-roots – rather like a zoo enclosure crossed with Angkor Wat – though eventually it will be replanted with the flowers so lovingly described in Country Life, and will be open to the public twice a year.

The house itself is a rambling hotchpotch of neo-Gothic and Classical, in red brick ornamented with stone detailing round the windows. Plasterboard ceilings have been torn down to reveal ornate cornices. One winding staircase leads up to a tiny door in a turret on the roof, which opens to reveal a playground of grand battlements and chimneys, perfect for roof-terrace parties. The higgledy-piggledy layout means no two flats will be the same.

The interiors will be modern, as Oakdene has decreed that buyers would be turned off by twee Victoriana. A converted stable block, in which flats cost up to £850,000, is the most contemporary in design: bathrooms are wired for sound and shiny kitchens are stuffed with appliances and studded with spotlights.

Nine flats have been sold so far, and the rest will be released over the next four months. Much care has gone into the showflat in the main building, yet it is in some ways the least impressive plot, suffering from a lack of external walls and thus windows. The decor is faux-luxurious, a style that seems to appeal to the tax-exile target market. “We’ve had couples who live in Dubai, Barbados and Guernsey and will use this as their foothold in England”, says Massingham. These buyers are “downsizing couples who don’t want a big five-bedroom home to look after any more.” Thehouse will, it seems, be used primarily as a weekend retreat rather than a home.

It is to be hoped that the jetset buyers will get on well with each other on those busy weekends, because privacy is not the development’s finest point: you are very close to your neighbours. The attractions, however, are great – the history, the architecture and gardens, the rural setting a 30-minute train ride from London, the proximity of Penshurst with its shop and pubs, and the security of high walls. Even the show-off aspect is in keeping with the original spirit of the house, built with City money to entertain the celebrities of the day. George Drummond, one feels, would have approved.

Contact: 0800 032 9650; www.swaylands.co.uk

Times recommends

• Second homes: credit crunch casualty or money-spinner?
• How I built my own sliding house
• Top 10 incentives for house buyers