Will Alsop OBE, the maverick architect and member of the Royal Academy (he is one of the selectors of this year’s Summer Exhibition), has won many awards over his 61 years. There is one, however, that has so far eluded him: best front garden of the year in Sheringham, north Norfolk, where he and his wife, Sheila, 62, have had a holiday home for more than 25 years. “It would mean more to me than the Stirling prize,” he has said.

Alsop is best known for modernist buildings such as Peckham library, in south London, where the upper floors are supported on spindly legs that look like skewwhiff knitting needles — for which he bagged the said Stirling prize in 2000. One of his most high-profile works in progress is Chips, a residential building in Manchester that resembles a neatly stacked pile of pommes frites.

So, it is something of a surprise to walk down the stone-set path of the long front garden in Norfolk, under the shade of a Giverny-style arbour, and arrive at a late-Victorian red-brick house filled with comfortable sofas, books, antique china and paintings. Where are the contemporary building materials, the modernist furniture, the wacky colours?

“It is not just my house, it is Sheila’s as well. I concur with her taste,” says the man sometimes known as Mr Blobby — surely for the splashes of colour on his buildings, rather than his cheerfully unkempt appearance, which comes wreathed in a fug of cigarette smoke. “This is not a great art; the house has nothing to do with great design. Using a word that most architects don’t use, it is ‘cosy’, and I am comfortable with that. Actually, as I get older, I try to incorporate that in my other work.”

We sit in the glass- and aluminium-walled dining room that Alsop added to the back of the house with materials left over from a project in France. The room, which replaced a conservatory and small terrace, is contemporary in feel, but hardly pushes the frontiers of design.

Beneath, in a series of terraces, is a narrow jungle of a garden, planted with tree ferns, phormiums, palms and large pots of hostas and hydrangeas. You can just see the sea through the trees and over the top of the boundary wall. Goldfish swim in the small pond beneath the window.

There’s a lunchtime bottle of red on the long maple table, which Alsop designed. He points to a sofa at the end of the room that catches the evening light. “If you really analyse both the house and garden, it is all about different places to sit,” he says. “I like a bit of sitting. That’s the gin-and-tonic sofa.”

Another favourite G&T spot is an outdoor bench by the front door that benefits from the warmth captured by the red-brick wall behind it. Similar seats are dotted around the jungly garden for different parts of the day. There is even one by the barbecue, though, he says, “we try not to have barbecues — dangerous things. They are the quickest way to arguments”. His role is to light the thing and eat the results: “If I cook, it is always burnt.”

The couple bought the five-bedroom house in 1982. They wanted somewhere out of London for weekends and holidays (these come less frequently nowadays, as Alsop is often abroad on business; he has offices in Shanghai, Beijing, Toronto and Singapore, and is a professor of architecture in Vienna), and Sheila’s parents lived nearby.

It is also a six-minute walk from the station, which was convenient, as Sheila didn’t drive at the time. When their children, Oliver, now 31, Nancy, 29, and Piers, 24, were growing up, friends could be easily collected from the station.

Queen Victoria’s visits to the nearby Sandringham estate, which she bought in 1863, popularised the area, but Sheringham is now a rather down-at-heel, old-fashioned kind of seaside resort. Although Alsop lives in Kensington, west London, he has no desire to join those neighbours from the capital who decamp in clannish clumps to smarter towns such as Burnham Market and Wells-next-the-Sea, to the west.

The town has several run-down hotels, a legacy from its Victorian glory days. “If someone wanted to invest, you could do something fantastic with these hotels, but they don’t, so that is the end of it,” says Alsop. His own house was once the stables for such a hotel; in the second world war, it was used by the military for maintaining motor vehicles. A local builder converted it in 1951, and it was pretty much unchanged when the Alsops bought it 30 years later from the estate of an old lady who had died. “The house had been empty for six or nine months. Houses show that; you feel it when you go into them.”

Explore Gardens

• Hampton Court Palace Flower Show

• Chelsea Flower Show

Times recommends

• Second homes: credit crunch casualty or money-spinner?
• How I built my own sliding house
• Top 10 incentives for house buyers