Rishi Sunak has spoken of the pain that racism caused him when he was growing up, as he said he believed the cricketing authorities were committed to stamping out widespread discrimination in the game.

Speaking to the BBC’s Test Match Special during the Ashes Test at Lord’s, the prime minister said he felt the “sting” of racism while growing up. He said his children were less likely to be exposed to racist abuse because of the progress the country had made in tackling it.

This week a report found that there was “widespread and deep-rooted” racism, sexism, elitism and class-based bias at all levels of cricket. The 317-page report from the Independent Commission for Equity in Cricket (Icec) warned that “it’s not banter or just a few bad apples” that were causing the problems.

Sunak, Britain’s first British-Asian prime minister, said the report had made him sad but he was reassured that the England and Wales Cricket Board was “absolutely committed to fixing all the problems”.

In the interview with the BBC cricket correspondent Jonathan Agnew, he reflected on his own experiences of racism. “Of course I have experienced racism, growing up, in particular, and of course I know it exists,” he said. “It stings you in a way that very few other things do.”

He recalled one confrontation when he had been out in Southampton, where he grew up, with his younger brother and sister and they received racist abuse. “Some people said a bunch of things and I felt doubly bad … I had my younger brother and sister with me and I didn’t want them to hear it and be exposed to it. It was really hard,” he said.

“One thing I take comfort from is I think the things that happened to me when I was a kid, I think they wouldn’t happen to my kids today, because I think we’ve just made incredible progress as a country and we should be proud of that.”

After taking over the Tory leadership after the divisive premierships of Boris Johnson and Liz Truss, Sunak conceded that he had “the worst in-tray you can possibly have”, with inflation high and mortgage costs going up for millions.

Hinting at the job he had to do to improve Tory morale, he said he admired the leadership of the England cricket captain, Ben Stokes, who had taken a team that had suffered a series of losses and turned it around with “essentially the same players with a different approach”. He said: “As somebody who wants to lead a different way, it was just an interesting case study.”

Sunak said his best day as prime minister came when he was able to play cricket for half an hour in the Downing Street garden with the England team.