Should any of Mr Cameron's Notting Hill set of trendy Tories feel the need to cultivate a more statesman-like image than that projected by the celebrity glitz of W11, a move to a respectable bit of suburbia, somewhere with an impeccable libertarian Tory heritage, might just do the trick.

Ferry House, for sale for £3.75 million, is a ten-bedroom family home and former political retreat on a secluded bend of the Thames next to Syon Park. It was the home of Lord Gilmour of Craigmillar, otherwise known as Sir Ian Gilmour MP, editor and proprietor of The Spectator in the 1950s, Defence Secretary to Edward Heath and Lord Privy Seal in Margaret Thatcher's first government. A passionate social liberal, he was sacked by Mrs Thatcher in 1981 and remained a prominent critic of policies such as the poll tax. A baronet in his own right, he was made a life peer by John Major in 1992.

Before his death last year, Lord Gilmour had lived at Ferry House for more than 50 years with his wife, Lady Caroline Montagu-Douglas-Scott, whose sister was married to the 10th Duke of Northumberland, owner of the neighbouring Syon estate. She died in 2004. Ferry House, once a dower house on the estate and briefly home to J.M.W.Turner, was demolished during the Second World War after a run-in with a doodlebug. It was rebuilt in the Georgian style in 1953 and bought by the newly wedded Gilmours. According to a fawning Country Life in 2003, Lady Gilmour, a duke's daughter who had grown up in a Scottish castle, had been worried about living in London, so “a new home on another ducal estate...was a reassuring compromise”.

Today, Ferry House, which is connected to the neighbouring Rosery Cottage - together they stretch to 8,551sq ft - is being sold by the Gilmours' five children, who were brought up there but are now scattered across the globe. Christopher Gilmour, 52, the third son, is the restaurateur behind Christopher's in Covent Garden. Next month he is opening a new restaurant, Gilmour's, in Mayfair. “I have very happy childhood memories of the house,” he says. “It was very exciting to be brought up here with politics and politicians. My parents had an annual garden party and everybody of the era came - Ted Heath, Roy Jenkins...”

The London property market

Background

• Affordable homes in South West London

• The London property market

• The common calling

• The London Riviera

The house is in Old Isleworth (much nicer than Isleworth proper), on the edge of Syon Park, and looks across the Thames to Richmond Old Deer Park. The London Apprentice pub, which was frequented by Dickens, is near by. Inside, off the stone-flagged front hall, is a very dated plastic kitchen and a ramshackle chintzy living room crammed with mismatching floral sofas, rugs, antique mirrors, oil paintings, occasional tables piled high with books and a piano straining under the weight of an army of framed photos. Up the rather grand staircase, hung with oil paintings of the family, are an endless procession of bedrooms. Along the corridor that leads into Rosery Cottage are several more. But many are shabby, and the downstairs rooms in the cottage are laid out so as to obscure the back garden from view. As Christopher Gilmour says, it “needs imagination and a chequebook. It has had nothing done to it for 50 years.”

The benefit of a faux-Georgian building is that it is not listed, so a new owner could easily knock down some walls and build a conservatory.

The most unusual and intriguing aspect of the house is Lord Gilmour's library, built in the mid-1960s by the architect Philip Jebb. An octagonal white weatherboard “ivory tower”, it is accessed by a central spiral stone staircase hung with framed photographs of the 1979 Cabinet and archaic House of Lords' documents. The book-lined first-floor room has a cosy fireplace and views over the river and gardens.

The gardens were Lady Gilmour's passion. At the back of Ferry House a walled garden of just under an acre leads towards the river. The garden front of the house is classically symmetrical, with ivy and shrubs to soften the brick. The lawn is dotted with trees. A public path crosses between the bottom of the garden and the river, but is rarely used, according to Christopher, and is closed off with a gate. Next door is a beautiful pink-and-white Robert Adam pavilion owned by the Northumberland estate. The gardens are included with the freehold of the house, but for anyone keen on larger grounds, a ten-year lease on a further acre of woodland and lawns, which was leased to the Gilmours by the Syon estate, is available.

Christopher is not fussy about who buys the house, although one imagines that principled Tory politicians would be welcome. His one concern is the garden: “It would be a great pity to have my mother's rose bushes pulled out, but beggars can't be choosers!”

Knight Frank: 020-7591 8600

Savills: 020-7499 8644

Times recommends

• Fall in house sales hits builders, estate agents and mortgage brokers
• Why the property credit crunch favours cash buyers
• The blood-curdling history of Quarrelwood House