When I go to Kensington Palace Gardens today, I always notice a small door in one of the walls. When I was working with Diana, at the end of the day, she’d often say: “Ken, do you fancy going for a walk with the boys?” We’d get the key from the police, enter the public park, walk round the pond and end up playing football.

I first went to Kensington Palace in l986, to look after the security of the two princes. From the moment I met Diana, I knew I was working with a fairly normal, friendly person. Actually, we met in the White Room at Sandringham. William, then five, was making a terrible noise on the piano, while Harry, who was three, stood on a table rearranging a huge vase of lilies. “I don’t envy you looking after my sons,” Diana said. “They can be a ruddy nuisance.” William said, “No, I’m not a ruddy nuisance.” Harry chimed in – “I’m not either” – then fell off the table, breaking the priceless vase. Diana chased them out of the room, saying, “You see what I mean?” That was typical Diana, but her style was frowned on by the royal family, including her husband, who felt there should be a wider barrier between royals and the below-stairs staff.

Kensington Palace was divided into apartments for various royals, with Diana and Charles at the top of the pecking order. Although theirs was quite big, it was modest by royal standards, with a small suite of offices on the ground floor and the prince and princess’s private rooms, a dining room and a lounge on the first.

Diana’s rooms were modern and comfortable, in Laura Ashley style, with deep sofas and family pictures. The Prince of Wales’s rooms were more masculine, with lots of books and artefacts. From the beginning, Diana told me about Camilla Parker Bowles – make no mistake, she was always in the background. Charles spent most of his time at Highgrove [his country house in Gloucester-shire], but at that stage Diana seemed prepared to go along with it for the sake of the children. The nursery was quite big, with a kitchen and a play area. Next door was my room, which was very small, just a bed and a washbasin. The day would begin with a tremendous noise, because the boys were quite a handful. Diana would always go with them to their school, Wetherby, on Pembridge Square, in Notting Hill. Most parents would be in designer finery; the princess would be in her gym kit.

People think palaces have round-the-clock court jesters, but the opposite is true. By 6pm, the only staff left would be the butler, the chef, the dresser and me, the policeman. On many occasions, Diana said that to sit alone in Kensington Palace was like being locked up in a cell. “What are you doing for supper?” she’d say. The chef, Mervyn Wycherley, would cook us scrambled eggs or a soufflé, which we’d eat with a glass of wine in the kitchen. That’s how it was for six or seven years.

Diana wanted the princes to know what ordinary people did. One Saturday, when Charles was at Highgrove, she said, “Let’s take William and Harry in a taxi.” Off we went to Hyde Park Corner in a taxi, then to Piccadilly Circus by Tube. Harry and William wanted to go on a bus, so we took one to a restaurant called Smollensky’s Balloon, where we ate hamburgers.

I cannot recall an unhappy moment in my working life with Diana. My job went beyond the policeman role. Sometimes she asked me personal questions, such as what she should do about her affair with James Hewitt. There’s no policeman’s textbook on royal affairs, I said: “If you see the end coming, tell him it’s over.”

For me, I saw the end coming a year before I left. Kensington Palace became a different place. The princes were at school; the Prince of Wales was at Highgrove all the time. I left in October 1993. We’d just come back from Diana’s holiday in the Bahamas and she’d been to visit William at school, where she’d been very crusty with him. “This isn’t like you,” I said. She wanted to do some shopping on Kensington High Street, but instead of parking the car at Kensington Palace, as usual, she just ran off into the street. I thought, “I’m not going to run after you.” I knew she’d go to Tower Records, and eventually she came out, clearly distressed. I said, “This isn’t going to work any more.”

That December, Diana announced that she was withdrawing from public life. She could be angry and forceful, even spiteful at times, but she was also wonderful. The sight of that small door covered in wisteria always brings a lump to my throat. Interview by Ann McFerran Diana: A Journey in Words and Pictures by Ken Wharfe is available from EMP Publications; www.empgroup.co.uk

Times recommends

• Second homes: credit crunch casualty or money-spinner?
• How I built my own sliding house
• Top 10 incentives for house buyers