Indigenous leaders have lashed Peter Dutton's plan to no longer fly the Aboriginal or Torres Strait Islander flags at press conferences if he becomes Prime Minister, arguing he is seeking to divide people.

The Opposition Leader, who walked out of Prime Minister Kevin Rudd's apology to members of the Stolen Generations in 2008, told Sky News on Monday night, "we are a country united under one flag and if we're asking people to identify with different flags, no other country does that, and we are dividing our country unnecessarily".

"We are united as a country when we gather under one flag, which is what we should do on Australia Day," he said, reiterating his support for celebrating Invasion Day.

On Tuesday, Minister for Indigenous Australians, Malarndirri McCarthy, said Mr Dutton was proving himself unfit to be a future Prime Minister.

"Yet again, he's seeking to divide Australians and grab a few culture war headlines," Senator McCarthy said.

View this post on Instagram

A post shared by Senator Malarndirri McCarthy (@malarndirri)

The National Museum of Australia states: "Australia's national flags – the blue ensign, the Aboriginal flag and the Torres Strait Islander flag – are important symbols."

"The Australian Aboriginal Flag and Torres Strait Islander Flag were proclaimed flags of Australia under section 5 of the Flags Act on 14 July 1995," Senator McCarthy said.

"Aboriginal and Torres Strait Islander culture is the oldest continuing culture in the world, and I believe all Australians should take great pride in that."

Her predecessor, Linda Burney, was equally scathing, taking to social media to slam Mr Dutton.

"Peter Dutton is the man who walked out of the apology to the Stolen Generations," Ms Burney, who will retire at the next election, said.

"This is the man who gave no support to the referendum. He refused to offer bipartisan support on all Aboriginal and Torres Strait Islander social and justice issues. What else can you expect from a man like him?"

Mr Dutton has previously vowed to hold no truth-telling events under a government he leads, and despite saying he would help to lower Indigenous disadvantage, has not outlined plans, or costings, to do so.